[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:55 dist-test-slave-g8hp krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:55 dist-test-slave-g8hp krb5kdc[399](info): set up 2 sockets
Sep 29 14:41:55 dist-test-slave-g8hp krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 14:41:56.357016 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:56 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:56.484730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:56.598456 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:56.598778 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:56.598820 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:56.603058 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:56.603144 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:56.603163 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:56.603183 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:56.603196 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:56.608949 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:56.610096 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:56.611340 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:41:56.621318 412 server_base.cc:1034] running on GCE node
W20260929 14:41:56.621347 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:56.621297 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:56.621295 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:56.622212 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:56.623641 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:56.624853 412 hybrid_clock.cc:648] HybridClock initialized: now 1790692916624777 us; error 95 us; skew 500 ppm
Sep 29 14:41:56 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:56.628093 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:56.629232 412 webserver.cc:460] Webserver started at http://127.0.98.62:41855/ using document root <none> and password file <none>
I20260929 14:41:56.629803 412 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:56.629866 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:56.630107 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:56.632373 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "cb1b3be2fcc241199f5170a4fd25c11b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "0d72e1d1161162b26e12d56c51502fea"
server_key_iv: "7bb1295f11d50ca52b3c1d75c5b0cc70"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.632997 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb1b3be2fcc241199f5170a4fd25c11b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "0d72e1d1161162b26e12d56c51502fea"
server_key_iv: "7bb1295f11d50ca52b3c1d75c5b0cc70"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.637053 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:41:56.639767 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:56.641255 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:56.641382 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "cb1b3be2fcc241199f5170a4fd25c11b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "0d72e1d1161162b26e12d56c51502fea"
server_key_iv: "7bb1295f11d50ca52b3c1d75c5b0cc70"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.641492 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:56.657555 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:56.661203 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:56.661427 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:56.668679 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39815
I20260929 14:41:56.668727 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39815 every 8 connection(s)
I20260929 14:41:56.669739 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:56.672004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 412
I20260929 14:41:56.672192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:56.673463 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:56.673971 392 external_mini_cluster.cc:1428] Setting key 2758cbfb3c3b48984438ff467b7a05c0
I20260929 14:41:56.680601 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b: Bootstrap starting.
I20260929 14:41:56.683139 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:56.684075 480 log.cc:824] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:56.686394 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b: No bootstrap required, opened a new log
I20260929 14:41:56.690594 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 14:41:56.690917 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:56.690970 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1b3be2fcc241199f5170a4fd25c11b, State: Initialized, Role: FOLLOWER
I20260929 14:41:56.691442 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 14:41:56.691589 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:56.691658 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:56.691756 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:56.692878 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
I20260929 14:41:56.693249 480 leader_election.cc:302] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb1b3be2fcc241199f5170a4fd25c11b; no voters:
I20260929 14:41:56.693578 480 leader_election.cc:288] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:56.693709 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:56.693945 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [term 1 LEADER]: Becoming Leader. State: Replica: cb1b3be2fcc241199f5170a4fd25c11b, State: Running, Role: LEADER
I20260929 14:41:56.694324 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } }
Sep 29 14:41:56 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:56.694875 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:56.695706 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb1b3be2fcc241199f5170a4fd25c11b. Latest consensus state: current_term: 1 leader_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } } }
I20260929 14:41:56.695827 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:56.696218 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:56.697268 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb1b3be2fcc241199f5170a4fd25c11b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39815 } } }
I20260929 14:41:56.697373 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:56.699715 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:56.704578 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:56.678284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.0.98.62:39815)
0929 14:41:56.678844 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:56.678865 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:56.679964 (+ 1099us) server_negotiation.cc:407] Connection header received
0929 14:41:56.683516 (+ 3552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:56.683538 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:56.683988 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:56.684362 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:56.686938 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.688497 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.689923 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.690637 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.690811 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:56.695670 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:56.695678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:56.695699 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:56.695743 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:56.698192 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:56.699079 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:56.699086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:56.699092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:56.699382 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:56.700546 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:56.700549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:56.700551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:56.701266 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:56.701467 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:56.703296 (+ 1829us) server_negotiation.cc:299] Negotiation successful
0929 14:41:56.703542 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":176,"threads_started":1}
I20260929 14:41:56.710379 494 catalog_manager.cc:1347] Generated new cluster ID: 3509457e2d3642ccb3769289bc5182d7
I20260929 14:41:56.710472 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:56.728852 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:56.729871 494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:56.734848 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb1b3be2fcc241199f5170a4fd25c11b: Generated new TSK 0
I20260929 14:41:56.735579 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:41:56.769877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:56.913774 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:56.914091 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:56.914139 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:56.918294 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:56.918361 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:56.918424 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:41:56.923878 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:56.925010 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:56.926030 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:56.936368 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:56.936396 506 server_base.cc:1034] running on GCE node
W20260929 14:41:56.936331 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:56.936322 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:56.937173 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:56.937971 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:56.939164 506 hybrid_clock.cc:648] HybridClock initialized: now 1790692916939144 us; error 42 us; skew 500 ppm
Sep 29 14:41:56 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:56.942410 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:41:56.943445 506 webserver.cc:460] Webserver started at http://127.0.98.1:35313/ using document root <none> and password file <none>
I20260929 14:41:56.944051 506 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:56.944124 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:56.944350 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:56.946771 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "100922fab94e4074b0f3aceb50a4cf4b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "2bf430c7b4f5f3e32e40355de5fdf3dc"
server_key_iv: "ee34cfd5e0ef52389ededb81180b6fc6"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.947399 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "100922fab94e4074b0f3aceb50a4cf4b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "2bf430c7b4f5f3e32e40355de5fdf3dc"
server_key_iv: "ee34cfd5e0ef52389ededb81180b6fc6"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.951606 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:41:56.954339 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:56.955770 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:41:56.955895 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "100922fab94e4074b0f3aceb50a4cf4b"
format_stamp: "Formatted at 2026-09-29 14:41:56 on dist-test-slave-g8hp"
server_key: "2bf430c7b4f5f3e32e40355de5fdf3dc"
server_key_iv: "ee34cfd5e0ef52389ededb81180b6fc6"
server_key_version: "encryptionkey@0"
I20260929 14:41:56.956005 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:56.970371 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:56.974341 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:56.974665 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:56.975347 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:41:56.976248 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:41:56.976303 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:56.976357 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:41:56.976384 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:56.985788 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35207
I20260929 14:41:56.985806 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35207 every 8 connection(s)
I20260929 14:41:56.986802 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:41:56.987720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 506
I20260929 14:41:56.987830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:41:56.988090 392 external_mini_cluster.cc:1428] Setting key 01de1aed9edfd9c9046a1f77cfd7d9f6
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 29 14:41:56 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:57.006090 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:56.989013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60491 (local address 127.0.98.62:39815)
0929 14:41:56.989145 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:56.989148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:56.990254 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 14:41:56.991663 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:56.991667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:56.991727 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:56.991822 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:56.993259 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.994125 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.995554 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.995955 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.996071 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.001535 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:57.001538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:57.001541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:57.001573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:57.003282 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.004059 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.004064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.004065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.004173 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.004677 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.004681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.004684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.005016 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:57.005143 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:57.005739 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 14:41:57.005892 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:41:57.006956 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:56.989561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.1:60491)
0929 14:41:56.990074 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:56.990131 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:56.991426 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:56.992054 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:56.992062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:56.992454 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:56.993087 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:56.993107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.994293 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.994296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:56.995419 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:56.995431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:56.996120 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:56.996123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:56.996299 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:56.997018 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:56.997036 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:57.001291 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:57.003483 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.003490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.003506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.003924 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.004308 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.004312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.004313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.004540 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.005127 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:57.005134 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:57.005460 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 14:41:57.005634 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:41:57.006006 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
I20260929 14:41:57.008373 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 14:41:57.008689 635 heartbeater.cc:464] Registering TS with master...
I20260929 14:41:57.009316 635 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
I20260929 14:41:57.011158 444 ts_manager.cc:194] Registered new tserver with Master: 100922fab94e4074b0f3aceb50a4cf4b (127.0.98.1:35207)
I20260929 14:41:57.012522 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:41:57.020673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:57.136365 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:57.136662 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:57.136706 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:57.140916 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:57.141001 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:57.141088 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:41:57.146922 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:57.148058 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:57.149176 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:57.159310 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:57.159442 640 server_base.cc:1034] running on GCE node
W20260929 14:41:57.159571 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:57.159296 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:57.160082 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:57.160924 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:57.162103 640 hybrid_clock.cc:648] HybridClock initialized: now 1790692917162080 us; error 45 us; skew 500 ppm
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:57.165167 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:41:57.166261 640 webserver.cc:460] Webserver started at http://127.0.98.2:41557/ using document root <none> and password file <none>
I20260929 14:41:57.166862 640 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:57.166929 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:57.167130 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:57.169360 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "84df61c659d4480aa82ae444ce307429"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "e59f6f9dcea5b6f71029c8545c4e35e1"
server_key_iv: "77ec2e4c6fe1311429512e299fb04c26"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.170017 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84df61c659d4480aa82ae444ce307429"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "e59f6f9dcea5b6f71029c8545c4e35e1"
server_key_iv: "77ec2e4c6fe1311429512e299fb04c26"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.174141 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:41:57.177030 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.178618 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:57.178746 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "84df61c659d4480aa82ae444ce307429"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "e59f6f9dcea5b6f71029c8545c4e35e1"
server_key_iv: "77ec2e4c6fe1311429512e299fb04c26"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.178867 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:57.201220 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:57.205061 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:57.205261 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:57.205904 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:41:57.206806 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:41:57.206861 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.206914 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:41:57.206934 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.216881 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44111
I20260929 14:41:57.216958 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44111 every 8 connection(s)
I20260929 14:41:57.217926 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:41:57.227439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 640
I20260929 14:41:57.227635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:41:57.227916 392 external_mini_cluster.cc:1428] Setting key cfb545b7e48f9cdd3a03e27e76641fcb
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:57.233114 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.219892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56423 (local address 127.0.98.62:39815)
0929 14:41:57.220063 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:57.220068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:57.221183 (+ 1115us) server_negotiation.cc:407] Connection header received
0929 14:41:57.222668 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:57.222670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:57.222723 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:57.222803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:57.224174 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.224739 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.226113 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.226342 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.226440 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.229518 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:57.229522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:57.229524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:57.229550 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:57.230958 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.231530 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.231533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.231535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.231613 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.231957 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.231960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.231961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.232150 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:57.232233 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:57.232776 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 14:41:57.232949 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 14:41:57.233999 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.220363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.2:56423)
0929 14:41:57.220930 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:57.221043 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:57.222408 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:57.222988 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:57.222996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:57.223469 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:57.224000 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:57.224014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.224965 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.224969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:57.225970 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:57.225981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.226498 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.226501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:57.226674 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.227338 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:57.227362 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:57.229226 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:57.231085 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.231092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.231113 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.231411 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.231695 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.231698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.231700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.231862 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.232245 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:57.232253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:57.232532 (+ 279us) client_negotiation.cc:769] Sending connection context
0929 14:41:57.232704 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:41:57.233110 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":109,"threads_started":1}
I20260929 14:41:57.235765 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 14:41:57.236021 769 heartbeater.cc:464] Registering TS with master...
I20260929 14:41:57.236548 769 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:41:57.237751 444 ts_manager.cc:194] Registered new tserver with Master: 84df61c659d4480aa82ae444ce307429 (127.0.98.2:44111)
I20260929 14:41:57.238428 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:41:57.256976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:57.375902 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:57.376201 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:57.376246 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:57.380640 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:57.380738 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:57.380803 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:41:57.386425 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:57.387547 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:57.388707 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:41:57.399129 774 server_base.cc:1034] running on GCE node
W20260929 14:41:57.399060 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:57.399060 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:57.399070 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:57.399744 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:57.400401 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:57.401652 774 hybrid_clock.cc:648] HybridClock initialized: now 1790692917401622 us; error 48 us; skew 500 ppm
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:57.405086 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:41:57.406098 774 webserver.cc:460] Webserver started at http://127.0.98.3:39871/ using document root <none> and password file <none>
I20260929 14:41:57.406719 774 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:57.406791 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:57.407032 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:57.409265 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "97cdc6ddfe9d4458b9b28a264a236447"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "7a11b6ddcbfde2f9cade47b8aaeb3700"
server_key_iv: "d678f15d89d85f4abd5dfee45308b8e6"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.409849 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97cdc6ddfe9d4458b9b28a264a236447"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "7a11b6ddcbfde2f9cade47b8aaeb3700"
server_key_iv: "d678f15d89d85f4abd5dfee45308b8e6"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.414130 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:41:57.416769 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.418224 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:57.418341 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "97cdc6ddfe9d4458b9b28a264a236447"
format_stamp: "Formatted at 2026-09-29 14:41:57 on dist-test-slave-g8hp"
server_key: "7a11b6ddcbfde2f9cade47b8aaeb3700"
server_key_iv: "d678f15d89d85f4abd5dfee45308b8e6"
server_key_version: "encryptionkey@0"
I20260929 14:41:57.418444 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:57.442694 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:57.446313 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:57.446504 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:57.447115 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:41:57.447956 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:41:57.448011 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.448066 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:41:57.448096 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:57.458138 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43469
I20260929 14:41:57.458158 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43469 every 8 connection(s)
I20260929 14:41:57.459095 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:41:57.463943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 774
I20260929 14:41:57.464082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:41:57.464362 392 external_mini_cluster.cc:1428] Setting key 503b9cf7e1d7c8d3e0f46d9280c11d2a
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:57.474119 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.461225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35387 (local address 127.0.98.62:39815)
0929 14:41:57.461363 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:57.461366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:57.462312 (+ 946us) server_negotiation.cc:407] Connection header received
0929 14:41:57.463708 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:57.463712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:57.463763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:57.463842 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:57.465415 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.466073 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.467552 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.467771 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.467852 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.470510 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:57.470513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:57.470515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:57.470541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:57.471861 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.472361 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.472364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.472365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.472542 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.472951 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.472954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.472956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.473150 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:57.473232 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:57.473830 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 14:41:57.473964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:41:57.474947 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.461701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39815 (local address 127.0.98.3:35387)
0929 14:41:57.462157 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:57.462201 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:57.463458 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:57.464080 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:57.464091 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:57.464627 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:57.465262 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:57.465277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.466202 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.466206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:57.467443 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:57.467454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.467890 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.467893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:57.468044 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.468695 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:57.468715 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:57.470335 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:57.471976 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.471981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.471994 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.472253 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.472663 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:57.472666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:57.472667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:57.472821 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:57.473265 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:57.473271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:57.473540 (+ 269us) client_negotiation.cc:769] Sending connection context
0929 14:41:57.473757 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 14:41:57.474104 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260929 14:41:57.476037 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39815
I20260929 14:41:57.476265 903 heartbeater.cc:464] Registering TS with master...
I20260929 14:41:57.476827 903 heartbeater.cc:511] Master 127.0.98.62:39815 requested a full tablet report, sending...
I20260929 14:41:57.477970 444 ts_manager.cc:194] Registered new tserver with Master: 97cdc6ddfe9d4458b9b28a264a236447 (127.0.98.3:43469)
I20260929 14:41:57.478619 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35387
I20260929 14:41:57.479014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:41:57.506238 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.495975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.98.1:35207)
0929 14:41:57.496266 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:57.496279 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:57.496302 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:41:57.496377 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:57.496383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:57.496579 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:57.496697 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:57.497521 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.498322 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.499324 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.499813 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.499903 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.502512 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:57.502523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:57.502538 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:57.502569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:57.504313 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.504746 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.504752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.504753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.504850 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.505221 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.505225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.505226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.505499 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:57.505708 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:57.505900 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 14:41:57.506053 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
W20260929 14:41:57.507203 546 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:41:57.535779 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:57.524996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.0.98.1:35207)
0929 14:41:57.525153 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:57.525159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:57.525442 (+ 283us) server_negotiation.cc:407] Connection header received
0929 14:41:57.525624 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:57.525628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:57.525685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:57.525774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:57.526685 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.528160 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.529317 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:57.529563 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:57.529660 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:57.532580 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:57.532583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:57.532586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:57.532617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:57.533986 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.534507 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.534510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.534512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.534599 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:57.534903 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:57.534905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:57.534907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:57.535090 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:57.535198 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:57.535547 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 14:41:57.535655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:41:57.537068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 506
I20260929 14:41:57.542459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 640
I20260929 14:41:57.549662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 774
I20260929 14:41:57.554823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 412
2026-09-29T14:41:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2179 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[917](info): set up 2 sockets
Sep 29 14:41:57 dist-test-slave-g8hp krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 14:41:58.520464 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:41:58 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:41:58.644505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45489
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:58.758038 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:58.758327 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:58.758368 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:58.762481 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:41:58.762553 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:58.762571 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:58.762588 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:41:58.762600 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:41:58.768015 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:58.769170 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:58.770229 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:58.779069 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:58.779073 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:58.779188 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:58.779335 930 server_base.cc:1034] running on GCE node
I20260929 14:41:58.780047 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:58.781338 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:58.782528 930 hybrid_clock.cc:648] HybridClock initialized: now 1790692918782488 us; error 64 us; skew 500 ppm
Sep 29 14:41:58 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:58.791684 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:41:58.792762 930 webserver.cc:460] Webserver started at http://127.0.98.62:43613/ using document root <none> and password file <none>
I20260929 14:41:58.793402 930 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:58.793480 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:58.793725 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:58.795925 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "31f75d034ebb4139a9d9f76ef72a63e1"
format_stamp: "Formatted at 2026-09-29 14:41:58 on dist-test-slave-g8hp"
server_key: "b4d9367cb13672bf879a5ebbd1174dc1"
server_key_iv: "57464ad2db24999137a0912f4824029e"
server_key_version: "encryptionkey@0"
I20260929 14:41:58.796531 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "31f75d034ebb4139a9d9f76ef72a63e1"
format_stamp: "Formatted at 2026-09-29 14:41:58 on dist-test-slave-g8hp"
server_key: "b4d9367cb13672bf879a5ebbd1174dc1"
server_key_iv: "57464ad2db24999137a0912f4824029e"
server_key_version: "encryptionkey@0"
I20260929 14:41:58.800736 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:41:58.803452 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:58.804951 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:41:58.805094 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "31f75d034ebb4139a9d9f76ef72a63e1"
format_stamp: "Formatted at 2026-09-29 14:41:58 on dist-test-slave-g8hp"
server_key: "b4d9367cb13672bf879a5ebbd1174dc1"
server_key_iv: "57464ad2db24999137a0912f4824029e"
server_key_version: "encryptionkey@0"
I20260929 14:41:58.805203 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:58.827630 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:58.830793 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:58.830989 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:58.837896 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45489
I20260929 14:41:58.837908 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45489 every 8 connection(s)
I20260929 14:41:58.838863 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:41:58.841192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 930
I20260929 14:41:58.841285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:41:58.841513 392 external_mini_cluster.cc:1428] Setting key 9ef31c569b1c5895adb07491fb3d67eb
I20260929 14:41:58.841862 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:41:58.849067 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1: Bootstrap starting.
Sep 29 14:41:58 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:58.851855 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1: Neither blocks nor log segments found. Creating new log.
I20260929 14:41:58.852567 998 log.cc:824] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:41:58.854491 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1: No bootstrap required, opened a new log
I20260929 14:41:58.855914 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:58.842928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.0.98.62:45489)
0929 14:41:58.843354 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:58.843371 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:58.843408 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:41:58.844122 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:58.844140 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:58.844489 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:58.844778 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:58.845726 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:58.846804 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:58.847962 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:58.848461 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:58.848578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:58.851461 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:58.851466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:58.851481 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:58.851513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:58.853193 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:58.853618 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:58.853624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:58.853629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:58.853738 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:58.854002 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:58.854005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:58.854007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:58.854406 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:58.854581 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:58.854861 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 14:41:58.855155 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":130,"threads_started":1}
I20260929 14:41:58.858625 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260929 14:41:58.858884 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:41:58.858950 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f75d034ebb4139a9d9f76ef72a63e1, State: Initialized, Role: FOLLOWER
I20260929 14:41:58.859385 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260929 14:41:58.859519 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:41:58.859583 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:41:58.859680 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:41:58.860770 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260929 14:41:58.861150 998 leader_election.cc:302] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f75d034ebb4139a9d9f76ef72a63e1; no voters:
I20260929 14:41:58.861474 998 leader_election.cc:288] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:41:58.861639 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:41:58.861853 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [term 1 LEADER]: Becoming Leader. State: Replica: 31f75d034ebb4139a9d9f76ef72a63e1, State: Running, Role: LEADER
I20260929 14:41:58.862154 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } }
I20260929 14:41:58.862687 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:41:58.863754 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } } }
I20260929 14:41:58.863907 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:58.864188 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f75d034ebb4139a9d9f76ef72a63e1. Latest consensus state: current_term: 1 leader_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f75d034ebb4139a9d9f76ef72a63e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45489 } } }
I20260929 14:41:58.864297 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:41:58.864841 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:41:58.867697 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:41:58.873975 1012 catalog_manager.cc:1347] Generated new cluster ID: dffb6742746f4676855bbc3b79170998
I20260929 14:41:58.874054 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:41:58.888360 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:41:58.889390 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:41:58.896253 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f75d034ebb4139a9d9f76ef72a63e1: Generated new TSK 0
I20260929 14:41:58.897258 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:41:58.937124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:41:59.054807 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:41:59.055119 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:41:59.055166 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:41:59.060307 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:41:59.060402 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:41:59.060469 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:41:59.066550 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:41:59.067806 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:41:59.068953 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:41:59.079612 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:59.079618 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:41:59.079612 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:41:59.079967 1024 server_base.cc:1034] running on GCE node
I20260929 14:41:59.080495 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:41:59.081312 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:41:59.082523 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790692919082501 us; error 35 us; skew 500 ppm
Sep 29 14:41:59 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:41:59.085592 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:41:59.086592 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34743/ using document root <none> and password file <none>
I20260929 14:41:59.087184 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 14:41:59.087251 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:41:59.087481 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:41:59.089993 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "30c64c36703143b9a58203b185b9524f"
format_stamp: "Formatted at 2026-09-29 14:41:59 on dist-test-slave-g8hp"
server_key: "d57e68c0d86f735a5f7e0bc92a743343"
server_key_iv: "9cac6007a238cd87f56c99d17fcda4f1"
server_key_version: "encryptionkey@0"
I20260929 14:41:59.090737 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30c64c36703143b9a58203b185b9524f"
format_stamp: "Formatted at 2026-09-29 14:41:59 on dist-test-slave-g8hp"
server_key: "d57e68c0d86f735a5f7e0bc92a743343"
server_key_iv: "9cac6007a238cd87f56c99d17fcda4f1"
server_key_version: "encryptionkey@0"
I20260929 14:41:59.095144 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:41:59.098824 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:59.100318 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:41:59.100522 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "30c64c36703143b9a58203b185b9524f"
format_stamp: "Formatted at 2026-09-29 14:41:59 on dist-test-slave-g8hp"
server_key: "d57e68c0d86f735a5f7e0bc92a743343"
server_key_iv: "9cac6007a238cd87f56c99d17fcda4f1"
server_key_version: "encryptionkey@0"
I20260929 14:41:59.100644 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:41:59.128413 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:41:59.131872 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:41:59.132042 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:41:59.132632 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:41:59.133720 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:41:59.133773 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:59.133817 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:41:59.133832 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:41:59.143414 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38545
I20260929 14:41:59.143481 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38545 every 8 connection(s)
I20260929 14:41:59.144464 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:41:59.154520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1024
I20260929 14:41:59.154632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:41:59.154873 392 external_mini_cluster.cc:1428] Setting key ff5442eaf2455970755421e3005e1969
Sep 29 14:41:59 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:41:59.161026 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:59.146512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60161 (local address 127.0.98.62:45489)
0929 14:41:59.146668 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:41:59.146673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:41:59.147686 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 14:41:59.149495 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:41:59.149499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:41:59.149554 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:41:59.149633 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:41:59.151310 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:59.151897 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:59.153233 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:41:59.153499 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:41:59.153610 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:59.156692 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:41:59.156697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:41:59.156700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:41:59.156729 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:41:59.158373 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:59.159078 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:59.159082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:59.159083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:59.159173 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:41:59.159605 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:41:59.159608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:41:59.159609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:41:59.159875 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:41:59.159968 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:41:59.160723 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 14:41:59.160885 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:41:59.162274 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:41:59.146962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45489 (local address 127.0.98.1:60161)
0929 14:41:59.147486 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 14:41:59.147549 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 14:41:59.149215 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:41:59.149872 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:41:59.149884 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:41:59.150391 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:41:59.151023 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:59.151038 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:59.152053 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:59.152056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:59.153073 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:41:59.153087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:41:59.153698 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:41:59.153702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:41:59.153892 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:41:59.154573 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:41:59.154592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:41:59.156449 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:41:59.158525 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:59.158534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:59.158556 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:59.158957 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:59.159272 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:41:59.159275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:41:59.159277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:41:59.159485 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:41:59.159987 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:41:59.159994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:41:59.160388 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 14:41:59.160629 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 14:41:59.161069 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
I20260929 14:41:59.163731 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45489
I20260929 14:41:59.164043 1153 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:41:59.164799 1153 heartbeater.cc:511] Master 127.0.98.62:45489 requested a full tablet report, sending...
I20260929 14:41:59.166512 962 ts_manager.cc:194] Registered new tserver with Master: 30c64c36703143b9a58203b185b9524f (127.0.98.1:38545)
I20260929 14:41:59.167694 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:00.170259 1153 heartbeater.cc:503] Master 127.0.98.62:45489 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260929 14:42:00.176244 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.020s user 0.000s sys 0.003s
I20260929 14:42:00.176791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:00.292508 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:00.292796 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:00.292896 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:00.297389 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:00.297482 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:00.297572 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:00.303972 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:00.305234 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:00.306308 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:00.316780 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:00.316884 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:00.316989 1158 server_base.cc:1034] running on GCE node
W20260929 14:42:00.316955 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:00.317528 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:00.318216 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:00.319418 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790692920319388 us; error 38 us; skew 500 ppm
Sep 29 14:42:00 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:00.322691 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:00.323818 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43153/ using document root <none> and password file <none>
I20260929 14:42:00.324422 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:00.324491 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:00.324733 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:00.327029 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "dbae4294ff5f4d3bbaf099042af12bc0"
format_stamp: "Formatted at 2026-09-29 14:42:00 on dist-test-slave-g8hp"
server_key: "62670beb08a8624912a70f524dbaccd8"
server_key_iv: "a2716ea402cd379c68796f6ae20eb563"
server_key_version: "encryptionkey@0"
I20260929 14:42:00.327641 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dbae4294ff5f4d3bbaf099042af12bc0"
format_stamp: "Formatted at 2026-09-29 14:42:00 on dist-test-slave-g8hp"
server_key: "62670beb08a8624912a70f524dbaccd8"
server_key_iv: "a2716ea402cd379c68796f6ae20eb563"
server_key_version: "encryptionkey@0"
I20260929 14:42:00.331827 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:00.334749 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:00.336298 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:00.336452 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "dbae4294ff5f4d3bbaf099042af12bc0"
format_stamp: "Formatted at 2026-09-29 14:42:00 on dist-test-slave-g8hp"
server_key: "62670beb08a8624912a70f524dbaccd8"
server_key_iv: "a2716ea402cd379c68796f6ae20eb563"
server_key_version: "encryptionkey@0"
I20260929 14:42:00.336571 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:00.352123 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:00.355834 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:00.356027 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:00.356657 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:00.357728 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:00.357800 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:00.357862 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:00.357893 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:00.367894 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45265
I20260929 14:42:00.367924 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45265 every 8 connection(s)
I20260929 14:42:00.368897 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:00.373908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1158
I20260929 14:42:00.374034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:00.374326 392 external_mini_cluster.cc:1428] Setting key 484d21c122824863388d25786790e6f2
Sep 29 14:42:00 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:00.386123 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:00.370933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60465 (local address 127.0.98.62:45489)
0929 14:42:00.371392 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:00.371396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:00.372060 (+ 664us) server_negotiation.cc:407] Connection header received
0929 14:42:00.373538 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:00.373541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:00.373593 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:00.373718 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:00.375420 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:00.376095 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:00.377880 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:00.378234 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:00.378453 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:00.381568 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:00.381573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:00.381576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:00.381613 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:00.383479 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:00.384103 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:00.384108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:00.384110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:00.384216 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:00.384684 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:00.384688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:00.384689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:00.384955 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:00.385090 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:00.385846 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 14:42:00.385978 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260929 14:42:00.386919 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:00.371403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45489 (local address 127.0.98.2:60465)
0929 14:42:00.371859 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:00.371906 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:00.373308 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:00.373947 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:00.373957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:00.374588 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:00.375191 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:00.375223 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:00.376258 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:00.376262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:00.377740 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:00.377754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:00.378540 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:00.378543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:00.378712 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:00.379426 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:00.379445 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:00.381335 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:00.383629 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:00.383635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:00.383662 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:00.383976 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:00.384352 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:00.384356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:00.384358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:00.384563 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:00.385208 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:00.385214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:00.385496 (+ 282us) client_negotiation.cc:769] Sending connection context
0929 14:42:00.385695 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 14:42:00.386014 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260929 14:42:00.388186 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45489
I20260929 14:42:00.388470 1287 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:00.389047 1287 heartbeater.cc:511] Master 127.0.98.62:45489 requested a full tablet report, sending...
I20260929 14:42:00.390316 962 ts_manager.cc:194] Registered new tserver with Master: dbae4294ff5f4d3bbaf099042af12bc0 (127.0.98.2:45265)
I20260929 14:42:00.390924 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:42:01.393375 1287 heartbeater.cc:503] Master 127.0.98.62:45489 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260929 14:42:01.438587 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.062s user 0.001s sys 0.003s
I20260929 14:42:01.439112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:01.560945 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:01.561241 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:01.561290 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:01.565758 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:01.565828 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:01.565900 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:01.571667 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45489
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:01.572809 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:01.573896 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:01.584000 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:01.583999 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:01.584054 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:01.584043 1293 server_base.cc:1034] running on GCE node
I20260929 14:42:01.584887 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:01.585618 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:01.586833 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790692921586750 us; error 77 us; skew 500 ppm
Sep 29 14:42:01 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:01.589895 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:01.590871 1293 webserver.cc:460] Webserver started at http://127.0.98.3:35285/ using document root <none> and password file <none>
I20260929 14:42:01.591442 1293 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:01.591511 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:01.591724 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:01.593935 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ce09315621542e7852ebf242d67ebf8"
format_stamp: "Formatted at 2026-09-29 14:42:01 on dist-test-slave-g8hp"
server_key: "8e24b6bb2a3535d6f2077c0b69e0f53a"
server_key_iv: "f4481150d7b1765559af8b944f52a3e1"
server_key_version: "encryptionkey@0"
I20260929 14:42:01.594511 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ce09315621542e7852ebf242d67ebf8"
format_stamp: "Formatted at 2026-09-29 14:42:01 on dist-test-slave-g8hp"
server_key: "8e24b6bb2a3535d6f2077c0b69e0f53a"
server_key_iv: "f4481150d7b1765559af8b944f52a3e1"
server_key_version: "encryptionkey@0"
I20260929 14:42:01.599347 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 14:42:01.602396 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:01.603930 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:01.604087 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "5ce09315621542e7852ebf242d67ebf8"
format_stamp: "Formatted at 2026-09-29 14:42:01 on dist-test-slave-g8hp"
server_key: "8e24b6bb2a3535d6f2077c0b69e0f53a"
server_key_iv: "f4481150d7b1765559af8b944f52a3e1"
server_key_version: "encryptionkey@0"
I20260929 14:42:01.604322 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:01.628484 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:01.632295 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:01.632484 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:01.633124 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:01.633986 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:01.634035 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:01.634089 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:01.634117 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:01.642940 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39965
I20260929 14:42:01.642967 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39965 every 8 connection(s)
I20260929 14:42:01.643857 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:01.646495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1293
I20260929 14:42:01.646616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:01.646855 392 external_mini_cluster.cc:1428] Setting key a40e9c91001f1ffcd82d562143cadf10
Sep 29 14:42:01 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:01.658905 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.645858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39053 (local address 127.0.98.62:45489)
0929 14:42:01.646209 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.646214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.646967 (+ 753us) server_negotiation.cc:407] Connection header received
0929 14:42:01.648459 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.648462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.648514 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.648612 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:01.649910 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.650481 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.651687 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.651939 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.652030 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.654794 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:01.654797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:01.654799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:01.654827 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:01.656544 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.657215 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.657218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.657219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.657308 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.657709 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.657713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.657714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.657929 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:01.658024 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.658576 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.658719 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260929 14:42:01.660336 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.646301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45489 (local address 127.0.98.3:39053)
0929 14:42:01.646786 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.646841 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.648264 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.648790 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.648798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.649224 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:01.649764 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.649779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.650594 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.650596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.651567 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.651576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.652119 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.652122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.652299 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.652918 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:01.652936 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:01.654584 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:01.656672 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:01.656678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:01.656693 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:01.657072 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:01.657397 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:01.657401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:01.657403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:01.657558 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:01.658032 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:01.658039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:01.658322 (+ 283us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.658503 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.658895 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260929 14:42:01.661660 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45489
I20260929 14:42:01.661935 1422 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:01.662588 1422 heartbeater.cc:511] Master 127.0.98.62:45489 requested a full tablet report, sending...
I20260929 14:42:01.663813 961 ts_manager.cc:194] Registered new tserver with Master: 5ce09315621542e7852ebf242d67ebf8 (127.0.98.3:39965)
I20260929 14:42:01.664479 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39053
I20260929 14:42:01.674360 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:42:01.687224 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.678344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.0.98.62:45489)
0929 14:42:01.678555 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.678560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.678709 (+ 149us) server_negotiation.cc:407] Connection header received
0929 14:42:01.678808 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.678812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.678861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.678931 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:01.679736 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.680522 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.681535 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.681883 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.682011 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.683707 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:01.683710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:01.683712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:01.683743 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:01.685345 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.685842 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.685845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.685846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.685937 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.686268 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.686272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.686273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.686546 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:01.686726 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.686917 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.687039 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:01.692853 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:01.696165 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:01.713258 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.706720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45265 (local address 127.0.0.1:42368)
0929 14:42:01.707556 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.707567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.707704 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.708527 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.708533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.708555 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:01.708666 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.708671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.709959 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.709962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.711493 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.711503 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.712640 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.712643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.712738 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.712773 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.712938 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.713101 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":114,"thread_start_us":91,"threads_started":1}
I20260929 14:42:01.713254 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.707069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.0.98.2:45265)
0929 14:42:01.707412 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.707418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.707647 (+ 229us) server_negotiation.cc:407] Connection header received
0929 14:42:01.707841 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.707847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.708073 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.708231 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:01.708780 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.709837 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.711626 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.712478 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.712600 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.712766 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.712977 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.713096 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260929 14:42:01.713431 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.707536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38545 (local address 127.0.0.1:59856)
0929 14:42:01.708341 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.708354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.708452 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.709118 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.709121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.709136 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:01.709217 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.709223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.710475 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.710478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.711766 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.711777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.712936 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.712940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.713128 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.713152 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.713218 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.713300 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":92,"thread_start_us":85,"threads_started":1}
I20260929 14:42:01.714177 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.706695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39965 (local address 127.0.0.1:35710)
0929 14:42:01.707443 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.707503 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.707639 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.708831 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.708834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.708852 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:01.708976 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.708984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.710344 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.710347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.711765 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.711777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.713823 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.713827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.713918 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.713941 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.713997 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.714066 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":73,"thread_start_us":125,"threads_started":1}
I20260929 14:42:01.714318 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.706840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.0.98.3:39965)
0929 14:42:01.707848 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.708314 (+ 466us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.708343 (+ 29us) server_negotiation.cc:407] Connection header received
0929 14:42:01.708461 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.708466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.708669 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.709115 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:01.709284 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.710188 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.712943 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.713710 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.713824 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.713992 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.714046 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.714140 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":136,"threads_started":1}
I20260929 14:42:01.715171 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.707934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.0.98.1:38545)
0929 14:42:01.708175 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.708182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.708431 (+ 249us) server_negotiation.cc:407] Connection header received
0929 14:42:01.708564 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.708569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.708722 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.708984 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:01.709351 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.710344 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.711895 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.712771 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.714481 (+ 1710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.714846 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.714904 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.715019 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260929 14:42:01.717358 1221 tablet_service.cc:1467] Processing CreateTablet for tablet dd4430eddcd64f238440c0411c9abfc0 (DEFAULT_TABLE table=test-table [id=81b638c4ba324db1863850d4b5ae1ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:01.717366 1356 tablet_service.cc:1467] Processing CreateTablet for tablet dd4430eddcd64f238440c0411c9abfc0 (DEFAULT_TABLE table=test-table [id=81b638c4ba324db1863850d4b5ae1ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:01.718123 1087 tablet_service.cc:1467] Processing CreateTablet for tablet dd4430eddcd64f238440c0411c9abfc0 (DEFAULT_TABLE table=test-table [id=81b638c4ba324db1863850d4b5ae1ee4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:01.718639 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4430eddcd64f238440c0411c9abfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:01.718664 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4430eddcd64f238440c0411c9abfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:01.719255 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4430eddcd64f238440c0411c9abfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:01.728214 1442 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Bootstrap starting.
I20260929 14:42:01.728652 1443 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Bootstrap starting.
I20260929 14:42:01.731251 1442 tablet_bootstrap.cc:654] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:01.731256 1443 tablet_bootstrap.cc:654] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:01.732297 1442 log.cc:824] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:01.732617 1443 log.cc:824] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:01.735018 1442 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: No bootstrap required, opened a new log
I20260929 14:42:01.735013 1443 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: No bootstrap required, opened a new log
I20260929 14:42:01.735373 1442 ts_tablet_manager.cc:1397] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:42:01.735373 1443 ts_tablet_manager.cc:1397] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:42:01.735368 1441 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Bootstrap starting.
I20260929 14:42:01.737666 1441 tablet_bootstrap.cc:654] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:01.738404 1441 log.cc:824] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:01.739696 1443 raft_consensus.cc:348] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.740036 1443 raft_consensus.cc:374] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:01.740135 1443 raft_consensus.cc:729] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce09315621542e7852ebf242d67ebf8, State: Initialized, Role: FOLLOWER
I20260929 14:42:01.739967 1442 raft_consensus.cc:348] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.740226 1442 raft_consensus.cc:374] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:01.740283 1442 raft_consensus.cc:729] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbae4294ff5f4d3bbaf099042af12bc0, State: Initialized, Role: FOLLOWER
I20260929 14:42:01.740736 1442 consensus_queue.cc:260] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.740736 1443 consensus_queue.cc:260] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.740921 1441 tablet_bootstrap.cc:492] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: No bootstrap required, opened a new log
I20260929 14:42:01.741246 1441 ts_tablet_manager.cc:1397] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:42:01.741842 1443 ts_tablet_manager.cc:1428] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 14:42:01.741833 1442 ts_tablet_manager.cc:1428] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260929 14:42:01.742213 1422 heartbeater.cc:503] Master 127.0.98.62:45489 was elected leader, sending a full tablet report...
I20260929 14:42:01.746539 1441 raft_consensus.cc:348] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.746865 1441 raft_consensus.cc:374] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:01.746937 1441 raft_consensus.cc:729] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c64c36703143b9a58203b185b9524f, State: Initialized, Role: FOLLOWER
I20260929 14:42:01.747362 1441 consensus_queue.cc:260] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.748378 1441 ts_tablet_manager.cc:1428] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:42:01.832290 1448 raft_consensus.cc:482] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:01.832581 1448 raft_consensus.cc:504] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.833992 1448 leader_election.cc:288] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbae4294ff5f4d3bbaf099042af12bc0 (127.0.98.2:45265), 30c64c36703143b9a58203b185b9524f (127.0.98.1:38545)
I20260929 14:42:01.838194 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.834538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35871 (local address 127.0.98.1:38545)
0929 14:42:01.834680 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.834684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.834700 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:42:01.834841 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.834843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.834887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.834950 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:01.835218 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.835762 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.836995 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.837621 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.837743 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.837823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.837962 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.838050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:01.838208 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.834539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57009 (local address 127.0.98.2:45265)
0929 14:42:01.834686 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.834690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.835057 (+ 367us) server_negotiation.cc:407] Connection header received
0929 14:42:01.835166 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.835169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.835211 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.835274 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:01.835558 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.836039 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.837107 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.837590 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.837680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.837753 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.837963 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.838074 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:42:01.838212 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.834399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38545 (local address 127.0.98.3:35871)
0929 14:42:01.834576 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.834594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.834717 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.834972 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.834974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.834993 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:01.835109 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.835115 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.835870 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.835874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.836832 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.836843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.837753 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.837757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.837853 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.837870 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.837938 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.838021 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":1}
I20260929 14:42:01.838223 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.834399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45265 (local address 127.0.98.3:57009)
0929 14:42:01.834953 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:01.834972 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:01.835070 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:01.835312 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:01.835315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:01.835330 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:01.835447 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.835452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.836144 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.836148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.836953 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:01.836969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.837723 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.837727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:01.837828 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.837853 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:42:01.837925 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 14:42:01.838012 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":219,"thread_start_us":120,"threads_started":1}
I20260929 14:42:01.839016 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4430eddcd64f238440c0411c9abfc0" candidate_uuid: "5ce09315621542e7852ebf242d67ebf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" is_pre_election: true
I20260929 14:42:01.839015 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4430eddcd64f238440c0411c9abfc0" candidate_uuid: "5ce09315621542e7852ebf242d67ebf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c64c36703143b9a58203b185b9524f" is_pre_election: true
I20260929 14:42:01.839381 1107 raft_consensus.cc:2454] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce09315621542e7852ebf242d67ebf8 in term 0.
I20260929 14:42:01.839375 1241 raft_consensus.cc:2454] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce09315621542e7852ebf242d67ebf8 in term 0.
I20260929 14:42:01.840011 1311 leader_election.cc:302] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce09315621542e7852ebf242d67ebf8, dbae4294ff5f4d3bbaf099042af12bc0; no voters:
I20260929 14:42:01.840314 1448 raft_consensus.cc:2781] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:01.840408 1448 raft_consensus.cc:482] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:01.840457 1448 raft_consensus.cc:3037] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:01.845265 1448 raft_consensus.cc:504] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.845779 1448 leader_election.cc:288] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [CANDIDATE]: Term 1 election: Requested vote from peers dbae4294ff5f4d3bbaf099042af12bc0 (127.0.98.2:45265), 30c64c36703143b9a58203b185b9524f (127.0.98.1:38545)
I20260929 14:42:01.846352 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4430eddcd64f238440c0411c9abfc0" candidate_uuid: "5ce09315621542e7852ebf242d67ebf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbae4294ff5f4d3bbaf099042af12bc0"
I20260929 14:42:01.846516 1241 raft_consensus.cc:3037] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:01.847049 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4430eddcd64f238440c0411c9abfc0" candidate_uuid: "5ce09315621542e7852ebf242d67ebf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30c64c36703143b9a58203b185b9524f"
I20260929 14:42:01.847208 1107 raft_consensus.cc:3037] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:01.848234 1241 raft_consensus.cc:2454] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce09315621542e7852ebf242d67ebf8 in term 1.
I20260929 14:42:01.848785 1311 leader_election.cc:302] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce09315621542e7852ebf242d67ebf8, dbae4294ff5f4d3bbaf099042af12bc0; no voters:
I20260929 14:42:01.849129 1448 raft_consensus.cc:2781] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:01.849395 1448 raft_consensus.cc:686] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce09315621542e7852ebf242d67ebf8, State: Running, Role: LEADER
I20260929 14:42:01.849823 1448 consensus_queue.cc:237] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } }
I20260929 14:42:01.853814 962 catalog_manager.cc:5495] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce09315621542e7852ebf242d67ebf8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5ce09315621542e7852ebf242d67ebf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce09315621542e7852ebf242d67ebf8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbae4294ff5f4d3bbaf099042af12bc0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30c64c36703143b9a58203b185b9524f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38545 } health_report { overall_health: UNKNOWN } } }
W20260929 14:42:01.871786 1288 tablet.cc:2369] T dd4430eddcd64f238440c0411c9abfc0 P dbae4294ff5f4d3bbaf099042af12bc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 14:42:01 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:01.889303 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.878045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.1:38545)
0929 14:42:01.878247 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.878252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.878387 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:42:01.878540 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.878543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.878589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.878671 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:01.879656 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.880173 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.881059 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.881337 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.881441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.884634 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:01.884648 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:01.884665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:01.884699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:01.887179 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.887709 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.887716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.887717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.887813 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.888209 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.888212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.888213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.888511 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:01.888693 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.889029 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.889163 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260929 14:42:01.894846 1423 tablet.cc:2369] T dd4430eddcd64f238440c0411c9abfc0 P 5ce09315621542e7852ebf242d67ebf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:01.897814 1154 tablet.cc:2369] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:01.906564 1107 raft_consensus.cc:2454] T dd4430eddcd64f238440c0411c9abfc0 P 30c64c36703143b9a58203b185b9524f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce09315621542e7852ebf242d67ebf8 in term 1.
Sep 29 14:42:01 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:42:01 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:01.999297 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:01.989721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.0.98.1:38545)
0929 14:42:01.989874 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:01.989878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:01.990059 (+ 181us) server_negotiation.cc:407] Connection header received
0929 14:42:01.990234 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:01.990237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:01.990289 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:01.990362 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:01.991247 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.991851 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.992766 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:01.993127 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:01.993235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:01.995957 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:01.995961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:01.995963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:01.995996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:01.997506 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.997977 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.997980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.997981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.998058 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:01.998435 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:01.998438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:01.998439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:01.998664 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:01.998769 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:01.999047 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 14:42:01.999167 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 14:42:02 dist-test-slave-g8hp krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:42:02 dist-test-slave-g8hp krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:02.101058 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:02.090542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.98.1:38545)
0929 14:42:02.090703 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:02.090708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:02.090853 (+ 145us) server_negotiation.cc:407] Connection header received
0929 14:42:02.091012 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:02.091015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:02.091064 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:02.091145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:02.091908 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:02.092407 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:02.093314 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:02.093529 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:02.093630 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:02.097238 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:02.097242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:02.097244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:02.097277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:02.099048 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:02.099577 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:02.099580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:02.099581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:02.099678 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:02.099973 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:02.099977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:02.099979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:02.100318 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:02.100455 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:02.100732 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 14:42:02.100896 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 14:42:02.102520 1067 tablet_service.cc:2908] Rejecting scan request for tablet dd4430eddcd64f238440c0411c9abfc0: Uninitialized: clean time has not yet been initialized
I20260929 14:42:02.105012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1024
I20260929 14:42:02.111152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1158
I20260929 14:42:02.116562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1293
I20260929 14:42:02.121896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 930
2026-09-29T14:42:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4567 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:03 dist-test-slave-g8hp krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:03 dist-test-slave-g8hp krb5kdc[1465](info): set up 2 sockets
Sep 29 14:42:03 dist-test-slave-g8hp krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 14:42:04.502434 392 mini_kdc.cc:121] Time spent starting KDC: real 2.358s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 14:42:04.710927 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.208s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:04 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:04.820159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:04.934276 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:04.934566 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:04.934607 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:04.938725 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:04.938802 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:42:04.938817 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:04.938835 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:04.938849 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:04.938863 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:04.944428 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:04.945551 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:04.946554 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:04.956128 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:04.956262 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:04.956272 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:04.956413 1478 server_base.cc:1034] running on GCE node
I20260929 14:42:04.957326 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:04.958559 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:04.959745 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790692924959716 us; error 49 us; skew 500 ppm
Sep 29 14:42:04 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:04.970871 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:04.972033 1478 webserver.cc:460] Webserver started at http://127.0.98.62:36781/ using document root <none> and password file <none>
I20260929 14:42:04.972647 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:04.972718 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:04.972966 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:04.975436 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "28550cc1e9f14a1abdcfeb314b8f2152"
format_stamp: "Formatted at 2026-09-29 14:42:04 on dist-test-slave-g8hp"
server_key: "d7ef663b67f69c41fe8346beed8f1a9c"
server_key_iv: "08796ee14702c57d19d5013a8e9ad606"
server_key_version: "encryptionkey@0"
I20260929 14:42:04.976087 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "28550cc1e9f14a1abdcfeb314b8f2152"
format_stamp: "Formatted at 2026-09-29 14:42:04 on dist-test-slave-g8hp"
server_key: "d7ef663b67f69c41fe8346beed8f1a9c"
server_key_iv: "08796ee14702c57d19d5013a8e9ad606"
server_key_version: "encryptionkey@0"
I20260929 14:42:04.980265 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:04.983546 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:04.985357 1478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:04.985523 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "28550cc1e9f14a1abdcfeb314b8f2152"
format_stamp: "Formatted at 2026-09-29 14:42:04 on dist-test-slave-g8hp"
server_key: "d7ef663b67f69c41fe8346beed8f1a9c"
server_key_iv: "08796ee14702c57d19d5013a8e9ad606"
server_key_version: "encryptionkey@0"
I20260929 14:42:04.985658 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:05.010809 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:05.014294 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:05.014485 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:05.022588 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43611
I20260929 14:42:05.022696 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43611 every 8 connection(s)
I20260929 14:42:05.023655 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:05.026747 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:05.027215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1478
I20260929 14:42:05.027324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:05.027606 392 external_mini_cluster.cc:1428] Setting key fdc54c114ddcb66bd4a96c94c7a530b6
I20260929 14:42:05.034372 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Bootstrap starting.
I20260929 14:42:05.037163 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Neither blocks nor log segments found. Creating new log.
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:05.037923 1556 log.cc:824] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:05.040412 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: No bootstrap required, opened a new log
I20260929 14:42:05.043906 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.029085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.0.98.62:43611)
0929 14:42:05.029545 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.029560 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.029597 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:42:05.030335 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.030356 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.030755 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.031062 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.032065 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.033249 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.034431 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.035076 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.035203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.037927 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.037932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.037947 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.037978 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.040692 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.041219 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.041225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.041231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.041377 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.041633 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.041636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.041638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.042165 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.042350 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.042598 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.042910 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":127,"threads_started":1}
I20260929 14:42:05.044377 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260929 14:42:05.044664 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:05.044835 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28550cc1e9f14a1abdcfeb314b8f2152, State: Initialized, Role: FOLLOWER
I20260929 14:42:05.045318 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260929 14:42:05.045454 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:05.045516 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:05.045610 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:05.046674 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260929 14:42:05.046998 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28550cc1e9f14a1abdcfeb314b8f2152; no voters:
I20260929 14:42:05.047319 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:05.047457 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:05.047838 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [term 1 LEADER]: Becoming Leader. State: Replica: 28550cc1e9f14a1abdcfeb314b8f2152, State: Running, Role: LEADER
I20260929 14:42:05.048197 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } }
I20260929 14:42:05.048620 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:05.049768 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } } }
I20260929 14:42:05.049878 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:05.050042 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28550cc1e9f14a1abdcfeb314b8f2152. Latest consensus state: current_term: 1 leader_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28550cc1e9f14a1abdcfeb314b8f2152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43611 } } }
I20260929 14:42:05.050195 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:05.050585 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:05.053417 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:05.059803 1572 catalog_manager.cc:1347] Generated new cluster ID: df5961bfd0144e65981b883ac1938a5a
I20260929 14:42:05.059903 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:05.073264 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:05.074707 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:05.082281 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Generated new TSK 0
I20260929 14:42:05.083205 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:05.125191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:05.242416 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:05.242726 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:05.242774 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:05.247153 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:05.247241 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:05.247284 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:05.247319 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:05.253285 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:05.254408 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:05.255441 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:05.265545 1582 server_base.cc:1034] running on GCE node
W20260929 14:42:05.265588 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:05.265511 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:05.265726 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:05.266940 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:05.267673 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:05.268901 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790692925268871 us; error 43 us; skew 500 ppm
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:05.271880 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:05.272891 1582 webserver.cc:460] Webserver started at http://127.0.98.1:35919/ using document root <none> and password file <none>
I20260929 14:42:05.273607 1582 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:05.273689 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:05.273911 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:05.276378 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "697fea6392524ef3995b71449704f594"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ba4c4f4a15f60b1e2e4fc8b77889fd25"
server_key_iv: "7ef78f51807718172f8e04d087257a8f"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.277006 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "697fea6392524ef3995b71449704f594"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ba4c4f4a15f60b1e2e4fc8b77889fd25"
server_key_iv: "7ef78f51807718172f8e04d087257a8f"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.281026 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:05.283933 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.285450 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:05.285576 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "697fea6392524ef3995b71449704f594"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ba4c4f4a15f60b1e2e4fc8b77889fd25"
server_key_iv: "7ef78f51807718172f8e04d087257a8f"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.285691 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:05.305408 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:05.308689 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:05.308914 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:05.310384 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:05.310434 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.310489 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:05.310518 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:05.321120 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33525
I20260929 14:42:05.321512 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33525 every 8 connection(s)
I20260929 14:42:05.322249 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:05.324764 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.311385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54171 (local address 127.0.98.62:43611)
0929 14:42:05.311499 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.311503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.312645 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 14:42:05.313766 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.313770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.313841 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.313922 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.315276 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.315816 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.317600 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.317890 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.317994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.320947 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.320951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.320954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.320987 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.322494 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.323093 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.323096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.323098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.323174 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.323509 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.323513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.323515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.323756 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.323951 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.324449 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.324591 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:05.325585 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.311990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.1:54171)
0929 14:42:05.312483 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.312528 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.313538 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.314148 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.314156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.314531 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.315121 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.315137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.315947 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.315951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.317456 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.317468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.318052 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.318056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.318225 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.319005 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.319022 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.320696 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.322611 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.322617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.322632 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.322975 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.323261 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.323263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.323265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.323408 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.323954 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.323961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.324214 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.324390 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.324728 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260929 14:42:05.326834 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260929 14:42:05.326985 1713 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:05.327478 1713 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260929 14:42:05.329129 1508 ts_manager.cc:194] Registered new tserver with Master: 697fea6392524ef3995b71449704f594 (127.0.98.1:33525)
I20260929 14:42:05.330482 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54171
I20260929 14:42:05.332320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1582
I20260929 14:42:05.332430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:05.332695 392 external_mini_cluster.cc:1428] Setting key 906665603fdc21340465e29d52a3d70f
I20260929 14:42:05.336525 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.329112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.1:52553)
0929 14:42:05.329511 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.329522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.329606 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.329818 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.329820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.329988 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.330084 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.330088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.331045 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.331047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.331898 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.331909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.332407 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.332411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.332505 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.333100 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.333115 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.333555 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.335116 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.335120 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.335122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.335324 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.335575 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.335579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.335581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.335663 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.335991 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.335995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.336083 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.336175 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.336324 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":86,"threads_started":1}
I20260929 14:42:05.336519 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.329201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52553 (local address 127.0.98.62:43611)
0929 14:42:05.329314 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.329319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.329594 (+ 275us) server_negotiation.cc:407] Connection header received
0929 14:42:05.329670 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.329673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.329715 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.329805 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.330172 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.330895 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.332033 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.332272 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.332378 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.333690 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.333694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.333697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.333721 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.335012 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.335417 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.335421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.335423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.335495 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.335740 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.335743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.335745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.335907 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.335988 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.336252 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.336367 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:05.360776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:05.474220 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:05.474538 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:05.474602 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:05.478830 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:05.478910 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:05.478968 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:05.479027 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:05.484735 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:05.485918 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:05.487002 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:05.497098 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:05.497251 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:05.497094 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:05.497416 1722 server_base.cc:1034] running on GCE node
I20260929 14:42:05.497891 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:05.498557 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:05.499764 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790692925499736 us; error 40 us; skew 500 ppm
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:05.502790 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:05.503741 1722 webserver.cc:460] Webserver started at http://127.0.98.2:37289/ using document root <none> and password file <none>
I20260929 14:42:05.504279 1722 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:05.504343 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:05.504509 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:05.506647 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c624311af5e40e48c1f0a4766e08b35"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "34c8ac47ab2c5599098ba7cbff9f14eb"
server_key_iv: "eb4e7a099ecd31d08c60c824abe099fc"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.507200 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c624311af5e40e48c1f0a4766e08b35"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "34c8ac47ab2c5599098ba7cbff9f14eb"
server_key_iv: "eb4e7a099ecd31d08c60c824abe099fc"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.511164 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:05.513754 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.515226 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:05.515376 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "7c624311af5e40e48c1f0a4766e08b35"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "34c8ac47ab2c5599098ba7cbff9f14eb"
server_key_iv: "eb4e7a099ecd31d08c60c824abe099fc"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.515501 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:05.535805 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:05.539456 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:05.539645 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:05.541244 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:05.541303 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.541344 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:05.541360 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.552327 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37623
I20260929 14:42:05.552484 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37623 every 8 connection(s)
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:05.553694 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:05.557497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1722
I20260929 14:42:05.557600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:05.557878 392 external_mini_cluster.cc:1428] Setting key 1ee2866d81067fb323a18de1d5b53ec1
I20260929 14:42:05.557906 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.542295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35739 (local address 127.0.98.62:43611)
0929 14:42:05.542419 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.542422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.543791 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 14:42:05.545145 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.545149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.545210 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.545296 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.547114 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.547943 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.549987 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.550281 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.550395 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.554172 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.554176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.554178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.554208 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.555715 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.556267 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.556271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.556273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.556361 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.556693 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.556696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.556698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.556947 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.557035 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.557601 (+ 566us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.557734 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:42:05.558856 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.543081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.2:35739)
0929 14:42:05.543598 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.543670 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.544820 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.545523 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.545534 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.546088 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.546903 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.546929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.548112 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.548116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.549829 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.549844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.550461 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.550466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.551076 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.551781 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.551804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.553966 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.555847 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.555854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.555872 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.556142 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.556444 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.556447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.556449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.556596 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.557044 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.557051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.557336 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.557515 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.557883 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260929 14:42:05.561235 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260929 14:42:05.561437 1853 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:05.562021 1853 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260929 14:42:05.563138 1510 ts_manager.cc:194] Registered new tserver with Master: 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623)
I20260929 14:42:05.563889 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35739
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:42:05.575484 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.562637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.2:37887)
0929 14:42:05.568090 (+ 5453us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.568107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.568229 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.568559 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.568562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.568830 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.568990 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.568997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.569979 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.569982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.570670 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.570705 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.571217 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.571220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.571313 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.571881 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.571893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.572225 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.573915 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.573918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.573921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.574111 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.574363 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.574366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.574368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.574450 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.574950 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.574953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.575046 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.575130 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.575282 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5312,"thread_start_us":109,"threads_started":1}
I20260929 14:42:05.575479 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.562969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37887 (local address 127.0.98.62:43611)
0929 14:42:05.563074 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.563077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.568215 (+ 5138us) server_negotiation.cc:407] Connection header received
0929 14:42:05.568360 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.568377 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.568429 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.568550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.569114 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.569839 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.570830 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.571098 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.571186 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.572391 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.572394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.572396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.572420 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.573804 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.574198 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.574201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.574202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.574283 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.574572 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.574575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.574576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.574858 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.574947 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.575196 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.575318 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:05.589089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43611
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:05.706211 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:05.706498 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:05.706543 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:05.711179 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:05.711267 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:05.711310 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:05.711349 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:05.717216 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:05.718381 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:05.719538 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:05.729851 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:05.729797 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:05.729912 1862 server_base.cc:1034] running on GCE node
W20260929 14:42:05.729823 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:05.730830 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:05.731468 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:05.732695 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790692925732668 us; error 38 us; skew 500 ppm
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:05.735771 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:05.736750 1862 webserver.cc:460] Webserver started at http://127.0.98.3:39771/ using document root <none> and password file <none>
I20260929 14:42:05.737334 1862 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:05.737397 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:05.737588 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:05.740024 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "46c05c4d274347b5abc18704a0dcbe14"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ce0533a647246ebb95d31f2843f86edb"
server_key_iv: "8aeaaa6cc5d96779f7d7c09437b61bcf"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.740765 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "46c05c4d274347b5abc18704a0dcbe14"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ce0533a647246ebb95d31f2843f86edb"
server_key_iv: "8aeaaa6cc5d96779f7d7c09437b61bcf"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.744982 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:05.747563 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.749059 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:05.749198 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "46c05c4d274347b5abc18704a0dcbe14"
format_stamp: "Formatted at 2026-09-29 14:42:05 on dist-test-slave-g8hp"
server_key: "ce0533a647246ebb95d31f2843f86edb"
server_key_iv: "8aeaaa6cc5d96779f7d7c09437b61bcf"
server_key_version: "encryptionkey@0"
I20260929 14:42:05.749320 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:05.779755 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:05.783174 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:05.783365 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:05.784819 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:05.784897 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:05.784961 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:05.784991 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:05.795152 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34917
I20260929 14:42:05.795253 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34917 every 8 connection(s)
I20260929 14:42:05.796093 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:05.798512 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.785817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34019 (local address 127.0.98.62:43611)
0929 14:42:05.785954 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.785959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.786827 (+ 868us) server_negotiation.cc:407] Connection header received
0929 14:42:05.788220 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.788224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.788282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.788379 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.790189 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.790944 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.792179 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.792405 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.792490 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.795036 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.795040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.795043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.795072 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.796502 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.797083 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.797085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.797087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.797172 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.797471 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.797474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.797475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.797662 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.797737 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.798219 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.798386 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:42:05.799275 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.786151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.3:34019)
0929 14:42:05.786634 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.786695 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.787964 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.788570 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.788579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.789165 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.789992 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.790015 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.791076 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.791079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.792024 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.792033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.792545 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.792549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.792712 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.793399 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.793423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.794831 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.796627 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.796633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.796647 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.796986 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.797240 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.797241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.797243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.797390 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.797765 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.797770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.797995 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.798173 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.798429 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260929 14:42:05.800411 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43611
I20260929 14:42:05.800554 1993 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:05.801126 1993 heartbeater.cc:511] Master 127.0.98.62:43611 requested a full tablet report, sending...
I20260929 14:42:05.802201 1508 ts_manager.cc:194] Registered new tserver with Master: 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917)
I20260929 14:42:05.802917 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34019
I20260929 14:42:05.806345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 1862
I20260929 14:42:05.806456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:05.806738 392 external_mini_cluster.cc:1428] Setting key e42f198c6d0e4491bff9350269d244f1
I20260929 14:42:05.808792 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:42:05.809461 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.802696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42615 (local address 127.0.98.62:43611)
0929 14:42:05.802809 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.802814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.802914 (+ 100us) server_negotiation.cc:407] Connection header received
0929 14:42:05.803033 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.803036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.803074 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.803136 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.803559 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.804057 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.805100 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.805311 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.805382 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.806396 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.806400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.806403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.806427 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.807957 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.808326 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.808329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.808331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.808406 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.808632 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.808634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.808635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.808837 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.808955 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.809254 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.809345 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:42:05.809517 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.802556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.98.3:42615)
0929 14:42:05.802840 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.802853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.802945 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.803159 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.803162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.803345 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:05.803447 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.803454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.804177 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.804180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.805000 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.805009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.805389 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.805391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.805453 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.805912 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:05.805924 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:05.806268 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:05.808046 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.808049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.808052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.808244 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.808480 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:05.808483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:05.808485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:05.808558 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:05.808999 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:05.809002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:05.809092 (+ 90us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.809190 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.809344 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:42:05 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:05.836028 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.827169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.98.62:43611)
0929 14:42:05.827326 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.827330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.827445 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:42:05.827534 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.827536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.827588 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.827667 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:05.828535 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.829140 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.830141 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.830368 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.830458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.833017 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:05.833021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:05.833022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:05.833052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:05.834394 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.834846 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.834849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.834850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.834934 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:05.835176 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:05.835179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:05.835180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:05.835360 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:05.835454 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.835782 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.835898 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:42:05.839082 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:05.842324 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:05.859057 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.852332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34917 (local address 127.0.0.1:52070)
0929 14:42:05.853166 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.853222 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.853360 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.855090 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.855093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.855112 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:05.855205 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.855211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.856506 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.856509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.857674 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.857681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.858490 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.858493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.858605 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.858645 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.858757 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.858890 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":19,"thread_start_us":116,"threads_started":1}
I20260929 14:42:05.859066 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.852654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.98.3:34917)
0929 14:42:05.854684 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.854695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.854715 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:42:05.854784 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.854788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.854968 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.855106 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:05.855321 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.856382 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.857801 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.858367 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.858469 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.858645 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.858803 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.858905 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1918,"thread_start_us":101,"threads_started":1}
I20260929 14:42:05.859750 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.852257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33525 (local address 127.0.0.1:48630)
0929 14:42:05.853165 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.853276 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.853366 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.854352 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.854359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.854410 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:05.854556 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.854564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.856160 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.856163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.857646 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.857658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.859014 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.859018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.859108 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.859460 (+ 352us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.859525 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.859599 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":59,"thread_start_us":107,"threads_started":1}
I20260929 14:42:05.859889 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.853097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.0.1:39534)
0929 14:42:05.853580 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:05.853589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:05.853657 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:05.854671 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:05.854673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:05.854690 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:05.854762 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.854767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.856036 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.856039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.857661 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:05.857671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.859473 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.859477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:05.859564 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.859582 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:42:05.859645 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:42:05.859737 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260929 14:42:05.859979 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.853525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.98.2:37623)
0929 14:42:05.853807 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.853818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.853844 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:42:05.853922 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.853927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.854152 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.854325 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:05.854893 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.855920 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.857800 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.858623 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.858725 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.858899 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.859708 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.859818 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260929 14:42:05.860700 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:05.852455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.0.98.1:33525)
0929 14:42:05.853845 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:05.853850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:05.853869 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:42:05.853942 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:05.853946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:05.854170 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:05.854383 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:05.854885 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.856021 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.857798 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:05.858616 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:05.858834 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:05.858959 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:05.860456 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0929 14:42:05.860559 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":74,"threads_started":1}
I20260929 14:42:05.862317 1927 tablet_service.cc:1467] Processing CreateTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf (DEFAULT_TABLE table=test-table [id=4a54c4e147c6446ab760c387dc9af1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:05.862890 1787 tablet_service.cc:1467] Processing CreateTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf (DEFAULT_TABLE table=test-table [id=4a54c4e147c6446ab760c387dc9af1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:05.863368 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddda0d2d67c44f38b9a8d7174abe82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:05.863421 1647 tablet_service.cc:1467] Processing CreateTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf (DEFAULT_TABLE table=test-table [id=4a54c4e147c6446ab760c387dc9af1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:05.863955 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddda0d2d67c44f38b9a8d7174abe82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:05.864447 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddda0d2d67c44f38b9a8d7174abe82bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:05.870862 2016 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Bootstrap starting.
I20260929 14:42:05.871932 2018 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Bootstrap starting.
I20260929 14:42:05.873217 2017 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Bootstrap starting.
I20260929 14:42:05.873262 2016 tablet_bootstrap.cc:654] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:05.874054 2016 log.cc:824] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:05.874073 2018 tablet_bootstrap.cc:654] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:05.874905 2018 log.cc:824] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:05.875373 2017 tablet_bootstrap.cc:654] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:05.876164 2017 log.cc:824] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:05.876614 2016 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: No bootstrap required, opened a new log
I20260929 14:42:05.876827 2016 ts_tablet_manager.cc:1397] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:05.877247 2018 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: No bootstrap required, opened a new log
I20260929 14:42:05.877419 2018 ts_tablet_manager.cc:1397] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:42:05.879927 2016 raft_consensus.cc:348] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.880196 2016 raft_consensus.cc:374] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:05.880244 2016 raft_consensus.cc:729] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c05c4d274347b5abc18704a0dcbe14, State: Initialized, Role: FOLLOWER
I20260929 14:42:05.880422 2018 raft_consensus.cc:348] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.880684 2018 raft_consensus.cc:374] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:05.880749 2018 raft_consensus.cc:729] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c624311af5e40e48c1f0a4766e08b35, State: Initialized, Role: FOLLOWER
I20260929 14:42:05.880769 2016 consensus_queue.cc:260] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.881567 1993 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260929 14:42:05.881907 2016 ts_tablet_manager.cc:1428] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260929 14:42:05.881314 2018 consensus_queue.cc:260] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.882449 2017 tablet_bootstrap.cc:492] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: No bootstrap required, opened a new log
I20260929 14:42:05.882699 2017 ts_tablet_manager.cc:1397] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260929 14:42:05.883152 2018 ts_tablet_manager.cc:1428] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:42:05.883469 1853 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260929 14:42:05.887095 2017 raft_consensus.cc:348] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.887363 2017 raft_consensus.cc:374] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:05.887405 2017 raft_consensus.cc:729] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697fea6392524ef3995b71449704f594, State: Initialized, Role: FOLLOWER
I20260929 14:42:05.887924 2017 consensus_queue.cc:260] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:05.888600 1713 heartbeater.cc:503] Master 127.0.98.62:43611 was elected leader, sending a full tablet report...
I20260929 14:42:05.888854 2017 ts_tablet_manager.cc:1428] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 14:42:06.047138 1994 tablet.cc:2369] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:06.055351 1854 tablet.cc:2369] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:06.073745 1714 tablet.cc:2369] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:06.210525 2024 raft_consensus.cc:482] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:06.210777 2024 raft_consensus.cc:504] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:06.212049 2024 leader_election.cc:288] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917), 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623)
I20260929 14:42:06.217854 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.212487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52233 (local address 127.0.98.2:37623)
0929 14:42:06.212679 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.212682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.212927 (+ 245us) server_negotiation.cc:407] Connection header received
0929 14:42:06.213134 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.213137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.213186 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.213268 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:06.213895 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.214754 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.216497 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.217237 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.217343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.217412 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.217627 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.217712 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 14:42:06.217881 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.212353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.98.1:52233)
0929 14:42:06.212832 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.212843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.212985 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.213325 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.213328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.213354 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:06.213728 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.213735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.214859 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.214863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.216332 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.216346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.217368 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.217372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.217480 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.217510 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.217594 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.217679 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":54,"spinlock_wait_cycles":23680,"thread_start_us":97,"threads_started":1}
I20260929 14:42:06.217918 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.212461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48243 (local address 127.0.98.3:34917)
0929 14:42:06.212592 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.212597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.212911 (+ 314us) server_negotiation.cc:407] Connection header received
0929 14:42:06.213110 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.213114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.213180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.213268 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:06.213895 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.214705 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.216497 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.217220 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.217343 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.217412 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.217726 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.217813 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:06.218348 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.212320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34917 (local address 127.0.98.1:48243)
0929 14:42:06.212785 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.212802 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.212994 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.213325 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.213328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.213358 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:06.213725 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.213735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.214859 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.214863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.216336 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.216346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.217489 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.217492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.217584 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.217608 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.218094 (+ 486us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.218191 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":154,"spinlock_wait_cycles":17024,"thread_start_us":104,"threads_started":1}
I20260929 14:42:06.218716 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddda0d2d67c44f38b9a8d7174abe82bf" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35" is_pre_election: true
I20260929 14:42:06.219141 1807 raft_consensus.cc:2454] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 0.
I20260929 14:42:06.219658 1599 leader_election.cc:302] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697fea6392524ef3995b71449704f594, 7c624311af5e40e48c1f0a4766e08b35; no voters:
I20260929 14:42:06.219673 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddda0d2d67c44f38b9a8d7174abe82bf" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c05c4d274347b5abc18704a0dcbe14" is_pre_election: true
I20260929 14:42:06.220005 1947 raft_consensus.cc:2454] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 0.
I20260929 14:42:06.220036 2024 raft_consensus.cc:2781] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:06.220160 2024 raft_consensus.cc:482] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:06.220218 2024 raft_consensus.cc:3037] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.221380 2024 raft_consensus.cc:504] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:06.221839 2024 leader_election.cc:288] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 election: Requested vote from peers 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917), 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623)
I20260929 14:42:06.222254 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddda0d2d67c44f38b9a8d7174abe82bf" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c05c4d274347b5abc18704a0dcbe14"
I20260929 14:42:06.222371 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddda0d2d67c44f38b9a8d7174abe82bf" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35"
I20260929 14:42:06.222384 1947 raft_consensus.cc:3037] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.222534 1807 raft_consensus.cc:3037] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.223577 1807 raft_consensus.cc:2454] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 1.
I20260929 14:42:06.223904 1947 raft_consensus.cc:2454] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 1.
I20260929 14:42:06.223923 1599 leader_election.cc:302] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 697fea6392524ef3995b71449704f594, 7c624311af5e40e48c1f0a4766e08b35; no voters:
I20260929 14:42:06.224082 2024 raft_consensus.cc:2781] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:06.224303 2024 raft_consensus.cc:686] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 1 LEADER]: Becoming Leader. State: Replica: 697fea6392524ef3995b71449704f594, State: Running, Role: LEADER
I20260929 14:42:06.224685 2024 consensus_queue.cc:237] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } }
I20260929 14:42:06.228488 1509 catalog_manager.cc:5495] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 697fea6392524ef3995b71449704f594 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:06.321455 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.317297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.0.98.1:33525)
0929 14:42:06.317437 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.317440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.317456 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:42:06.317611 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.317613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.317675 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.317777 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:06.318057 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.318579 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.319623 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.319897 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.319986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.320415 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.321005 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.321099 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.321271 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.321334 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:42:06.328328 1807 raft_consensus.cc:1261] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Refusing update from remote peer 697fea6392524ef3995b71449704f594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.328328 1947 raft_consensus.cc:1261] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Refusing update from remote peer 697fea6392524ef3995b71449704f594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.329126 2024 consensus_queue.cc:1035] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.329408 2027 consensus_queue.cc:1035] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.337344 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334678226233024512 to 7334678225819217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334678226233024512}]
I20260929 14:42:06.338977 2031 mvcc.cc:205] Tried to move back new op lower bound from 7334678226233024512 to 7334678225819217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334678226233024512}]
I20260929 14:42:06.341316 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334678226233024512 to 7334678225819217920. Current Snapshot: MvccSnapshot[applied={T|T < 7334678226233024512 or (T in {7334678226233024512})}]
I20260929 14:42:06.349924 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:06.350111 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:06.352758 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Sending DeleteTablet for 3 replicas of tablet ddda0d2d67c44f38b9a8d7174abe82bf
I20260929 14:42:06.353673 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260929 14:42:06.353674 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52070
I20260929 14:42:06.353888 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet ddda0d2d67c44f38b9a8d7174abe82bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39534
I20260929 14:42:06.354377 2040 tablet_replica.cc:330] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: stopping tablet replica
I20260929 14:42:06.354713 2040 raft_consensus.cc:2229] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:06.354753 2042 tablet_replica.cc:330] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: stopping tablet replica
I20260929 14:42:06.355067 2042 raft_consensus.cc:2229] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:06.355171 2040 raft_consensus.cc:2258] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.355118 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:42:06.355410 2042 raft_consensus.cc:2258] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.355545 2041 tablet_replica.cc:330] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: stopping tablet replica
I20260929 14:42:06.355844 2041 raft_consensus.cc:2229] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 14:42:06.355865 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:06.356148 2041 raft_consensus.cc:2258] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.356820 2042 ts_tablet_manager.cc:1905] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.357584 2041 ts_tablet_manager.cc:1905] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.359916 2040 ts_tablet_manager.cc:1905] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.361472 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 (DEFAULT_TABLE table=test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:06.361491 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 (DEFAULT_TABLE table=test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:06.361745 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9c69f8a214bcb84c72b84017bc013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.361967 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9c69f8a214bcb84c72b84017bc013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.362562 2040 ts_tablet_manager.cc:1918] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:06.362654 2040 log.cc:1196] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/ddda0d2d67c44f38b9a8d7174abe82bf
I20260929 14:42:06.362943 2040 ts_tablet_manager.cc:1939] T ddda0d2d67c44f38b9a8d7174abe82bf P 697fea6392524ef3995b71449704f594: Deleting consensus metadata
I20260929 14:42:06.363983 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 (DEFAULT_TABLE table=test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:06.364264 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aa9c69f8a214bcb84c72b84017bc013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.364300 2017 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Bootstrap starting.
I20260929 14:42:06.365048 2041 ts_tablet_manager.cc:1918] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:06.365141 2041 log.cc:1196] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/ddda0d2d67c44f38b9a8d7174abe82bf
I20260929 14:42:06.365272 2017 tablet_bootstrap.cc:654] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.365442 2041 ts_tablet_manager.cc:1939] T ddda0d2d67c44f38b9a8d7174abe82bf P 46c05c4d274347b5abc18704a0dcbe14: Deleting consensus metadata
I20260929 14:42:06.366608 2016 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Bootstrap starting.
I20260929 14:42:06.366743 1497 catalog_manager.cc:4841] TS 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917): tablet ddda0d2d67c44f38b9a8d7174abe82bf (table test-table [id=4a54c4e147c6446ab760c387dc9af1f9]) successfully deleted
I20260929 14:42:06.366896 2018 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Bootstrap starting.
I20260929 14:42:06.367544 2016 tablet_bootstrap.cc:654] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.367771 2018 tablet_bootstrap.cc:654] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.368635 1495 catalog_manager.cc:4841] TS 697fea6392524ef3995b71449704f594 (127.0.98.1:33525): tablet ddda0d2d67c44f38b9a8d7174abe82bf (table test-table [id=4a54c4e147c6446ab760c387dc9af1f9]) successfully deleted
I20260929 14:42:06.368917 2042 ts_tablet_manager.cc:1918] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:06.369019 2042 log.cc:1196] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/ddda0d2d67c44f38b9a8d7174abe82bf
I20260929 14:42:06.369021 2016 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: No bootstrap required, opened a new log
I20260929 14:42:06.369095 2016 ts_tablet_manager.cc:1397] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 14:42:06.369567 2016 raft_consensus.cc:348] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.369694 2016 raft_consensus.cc:374] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.369726 2016 raft_consensus.cc:729] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c05c4d274347b5abc18704a0dcbe14, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.369742 2042 ts_tablet_manager.cc:1939] T ddda0d2d67c44f38b9a8d7174abe82bf P 7c624311af5e40e48c1f0a4766e08b35: Deleting consensus metadata
I20260929 14:42:06.369858 2016 consensus_queue.cc:260] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.370129 2016 ts_tablet_manager.cc:1428] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:06.371078 2018 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: No bootstrap required, opened a new log
I20260929 14:42:06.371073 1495 catalog_manager.cc:4841] TS 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623): tablet ddda0d2d67c44f38b9a8d7174abe82bf (table test-table [id=4a54c4e147c6446ab760c387dc9af1f9]) successfully deleted
I20260929 14:42:06.371172 2018 ts_tablet_manager.cc:1397] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 14:42:06.371625 2018 raft_consensus.cc:348] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.371745 2018 raft_consensus.cc:374] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.371776 2018 raft_consensus.cc:729] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c624311af5e40e48c1f0a4766e08b35, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.371899 2018 consensus_queue.cc:260] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.372128 2017 tablet_bootstrap.cc:492] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: No bootstrap required, opened a new log
I20260929 14:42:06.372252 2017 ts_tablet_manager.cc:1397] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 14:42:06.372359 2018 ts_tablet_manager.cc:1428] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:42:06.372934 2017 raft_consensus.cc:348] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.373067 2017 raft_consensus.cc:374] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.373103 2017 raft_consensus.cc:729] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697fea6392524ef3995b71449704f594, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.373221 2017 consensus_queue.cc:260] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.373497 2017 ts_tablet_manager.cc:1428] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:42:06.520948 2022 raft_consensus.cc:482] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:06.521097 2022 raft_consensus.cc:504] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.522148 2022 leader_election.cc:288] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 697fea6392524ef3995b71449704f594 (127.0.98.1:33525), 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623)
I20260929 14:42:06.528061 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.523160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33693 (local address 127.0.98.2:37623)
0929 14:42:06.523340 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.523346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.523362 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:42:06.523538 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.523541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.523594 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.523668 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:06.524142 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.524915 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.526759 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.527389 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.527501 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.527570 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.527824 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.527911 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 14:42:06.528244 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.523418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42607 (local address 127.0.98.1:33525)
0929 14:42:06.523606 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.523611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.523625 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:06.523682 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.523684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.523729 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.523802 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:06.524288 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.525011 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.526986 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.527676 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.527796 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.527870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.528032 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.528133 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 14:42:06.528517 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.522969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37623 (local address 127.0.98.3:33693)
0929 14:42:06.523281 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.523295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.523389 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.523794 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.523797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.523817 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:06.523938 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.523945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.525039 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.525043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.526632 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.526645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.527555 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.527560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.527686 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.527704 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.528269 (+ 565us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.528356 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":31,"thread_start_us":105,"threads_started":1}
I20260929 14:42:06.528802 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.522459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33525 (local address 127.0.98.3:42607)
0929 14:42:06.522782 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.522799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.522936 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.523994 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.523997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.524025 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:06.524147 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.524154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.525148 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.525151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.526523 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.526536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.527804 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.527807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.527896 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.527917 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.528472 (+ 555us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.528558 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260929 14:42:06.529497 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9c69f8a214bcb84c72b84017bc013" candidate_uuid: "46c05c4d274347b5abc18704a0dcbe14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35" is_pre_election: true
I20260929 14:42:06.529791 1807 raft_consensus.cc:2454] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c05c4d274347b5abc18704a0dcbe14 in term 0.
I20260929 14:42:06.529812 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9c69f8a214bcb84c72b84017bc013" candidate_uuid: "46c05c4d274347b5abc18704a0dcbe14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697fea6392524ef3995b71449704f594" is_pre_election: true
I20260929 14:42:06.530041 1667 raft_consensus.cc:2454] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46c05c4d274347b5abc18704a0dcbe14 in term 0.
I20260929 14:42:06.530519 1879 leader_election.cc:302] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c05c4d274347b5abc18704a0dcbe14, 7c624311af5e40e48c1f0a4766e08b35; no voters:
I20260929 14:42:06.531042 2022 raft_consensus.cc:2781] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:06.531109 2022 raft_consensus.cc:482] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:06.531165 2022 raft_consensus.cc:3037] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.532454 2022 raft_consensus.cc:504] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.532895 2022 leader_election.cc:288] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [CANDIDATE]: Term 1 election: Requested vote from peers 697fea6392524ef3995b71449704f594 (127.0.98.1:33525), 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623)
I20260929 14:42:06.533324 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9c69f8a214bcb84c72b84017bc013" candidate_uuid: "46c05c4d274347b5abc18704a0dcbe14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "697fea6392524ef3995b71449704f594"
I20260929 14:42:06.533464 1667 raft_consensus.cc:3037] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.533674 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aa9c69f8a214bcb84c72b84017bc013" candidate_uuid: "46c05c4d274347b5abc18704a0dcbe14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35"
I20260929 14:42:06.533808 1807 raft_consensus.cc:3037] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.534766 1667 raft_consensus.cc:2454] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c05c4d274347b5abc18704a0dcbe14 in term 1.
I20260929 14:42:06.534894 1807 raft_consensus.cc:2454] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46c05c4d274347b5abc18704a0dcbe14 in term 1.
I20260929 14:42:06.535504 1879 leader_election.cc:302] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c05c4d274347b5abc18704a0dcbe14, 7c624311af5e40e48c1f0a4766e08b35; no voters:
I20260929 14:42:06.535760 2022 raft_consensus.cc:2781] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:06.535979 2022 raft_consensus.cc:686] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 1 LEADER]: Becoming Leader. State: Replica: 46c05c4d274347b5abc18704a0dcbe14, State: Running, Role: LEADER
I20260929 14:42:06.536304 2022 consensus_queue.cc:237] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.539242 1508 catalog_manager.cc:5495] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "46c05c4d274347b5abc18704a0dcbe14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } health_report { overall_health: HEALTHY } } }
Sep 29 14:42:06 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:06.595567 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.586698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43611 (local address 127.0.0.1:37798)
0929 14:42:06.587007 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.587023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.587148 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.587379 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.587382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.587822 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:06.587922 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.587930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.588792 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.588795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.589557 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.589569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.590002 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.590005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.590098 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.590861 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:06.590875 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:06.592457 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:06.594099 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:06.594111 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:06.594117 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:06.594380 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:06.594649 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:06.594651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:06.594653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:06.594742 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:06.595074 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:06.595080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:06.595197 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.595288 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.595418 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260929 14:42:06.595675 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.586796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.0.98.62:43611)
0929 14:42:06.587069 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.587078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.587092 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:06.587236 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.587239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.587278 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.587347 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:06.588039 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.588674 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.589675 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.589887 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.589974 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.592627 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:06.592630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:06.592632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:06.592659 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:06.593977 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.594481 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.594484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.594486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.594569 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.594811 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.594813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.594814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.594999 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:06.595070 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.595381 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.595492 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":106,"threads_started":1}
I20260929 14:42:06.605894 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:42:06.606698 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:06.612052 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf1b6c471584ed8a71529bffc01b9ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=190f3f3c5d4d46a591df918b000756d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:06.612052 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf1b6c471584ed8a71529bffc01b9ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=190f3f3c5d4d46a591df918b000756d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:06.612052 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf1b6c471584ed8a71529bffc01b9ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=190f3f3c5d4d46a591df918b000756d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:06.612421 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf1b6c471584ed8a71529bffc01b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.612421 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf1b6c471584ed8a71529bffc01b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.612423 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf1b6c471584ed8a71529bffc01b9ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:06.614784 2017 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: Bootstrap starting.
I20260929 14:42:06.615156 2018 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: Bootstrap starting.
I20260929 14:42:06.615156 2016 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: Bootstrap starting.
I20260929 14:42:06.615679 2017 tablet_bootstrap.cc:654] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.616037 2016 tablet_bootstrap.cc:654] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.616032 2018 tablet_bootstrap.cc:654] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:06.617322 2017 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: No bootstrap required, opened a new log
I20260929 14:42:06.617321 2018 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: No bootstrap required, opened a new log
I20260929 14:42:06.617410 2018 ts_tablet_manager.cc:1397] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:06.617430 2017 ts_tablet_manager.cc:1397] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:06.617651 2016 tablet_bootstrap.cc:492] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: No bootstrap required, opened a new log
I20260929 14:42:06.617734 2016 ts_tablet_manager.cc:1397] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 14:42:06.617841 2018 raft_consensus.cc:348] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.617877 2017 raft_consensus.cc:348] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.617969 2017 raft_consensus.cc:374] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.617969 2018 raft_consensus.cc:374] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.618007 2018 raft_consensus.cc:729] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c624311af5e40e48c1f0a4766e08b35, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.618016 2017 raft_consensus.cc:729] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 697fea6392524ef3995b71449704f594, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.618126 2017 consensus_queue.cc:260] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.618126 2018 consensus_queue.cc:260] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.618151 2016 raft_consensus.cc:348] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.618247 2016 raft_consensus.cc:374] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:06.618280 2016 raft_consensus.cc:729] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c05c4d274347b5abc18704a0dcbe14, State: Initialized, Role: FOLLOWER
I20260929 14:42:06.618387 2017 ts_tablet_manager.cc:1428] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:06.618387 2018 ts_tablet_manager.cc:1428] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:06.618402 2016 consensus_queue.cc:260] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.618636 2016 ts_tablet_manager.cc:1428] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:06.618620 2023 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.618611 2036 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.618753 2036 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.618753 2023 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.618808 2022 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.618932 2022 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.770184 2036 raft_consensus.cc:482] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:06.770323 2036 raft_consensus.cc:504] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.770874 2036 leader_election.cc:288] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623), 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917)
I20260929 14:42:06.771291 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf1b6c471584ed8a71529bffc01b9ba" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35" is_pre_election: true
I20260929 14:42:06.771319 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf1b6c471584ed8a71529bffc01b9ba" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c05c4d274347b5abc18704a0dcbe14" is_pre_election: true
I20260929 14:42:06.771432 1807 raft_consensus.cc:2454] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 0.
I20260929 14:42:06.771462 1947 raft_consensus.cc:2454] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 0.
I20260929 14:42:06.771720 1601 leader_election.cc:302] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46c05c4d274347b5abc18704a0dcbe14, 697fea6392524ef3995b71449704f594, 7c624311af5e40e48c1f0a4766e08b35; no voters:
I20260929 14:42:06.771922 2036 raft_consensus.cc:2781] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:06.771986 2036 raft_consensus.cc:482] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:06.772028 2036 raft_consensus.cc:3037] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.773232 2036 raft_consensus.cc:504] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.773617 2036 leader_election.cc:288] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 election: Requested vote from peers 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623), 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917)
I20260929 14:42:06.774042 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf1b6c471584ed8a71529bffc01b9ba" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c624311af5e40e48c1f0a4766e08b35"
I20260929 14:42:06.774050 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf1b6c471584ed8a71529bffc01b9ba" candidate_uuid: "697fea6392524ef3995b71449704f594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46c05c4d274347b5abc18704a0dcbe14"
I20260929 14:42:06.774181 1947 raft_consensus.cc:3037] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.774187 1807 raft_consensus.cc:3037] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:06.775309 1947 raft_consensus.cc:2454] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 1.
I20260929 14:42:06.775309 1807 raft_consensus.cc:2454] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 697fea6392524ef3995b71449704f594 in term 1.
I20260929 14:42:06.775642 1601 leader_election.cc:302] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46c05c4d274347b5abc18704a0dcbe14, 697fea6392524ef3995b71449704f594; no voters:
I20260929 14:42:06.775835 2036 raft_consensus.cc:2781] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:06.775921 2036 raft_consensus.cc:686] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [term 1 LEADER]: Becoming Leader. State: Replica: 697fea6392524ef3995b71449704f594, State: Running, Role: LEADER
I20260929 14:42:06.776078 2036 consensus_queue.cc:237] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } }
I20260929 14:42:06.776777 2027 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: TxnStatusTablet state changed. Reason: New leader 697fea6392524ef3995b71449704f594. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.776907 2027 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:06.777237 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:42:06.777417 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:42:06.777933 1510 catalog_manager.cc:5495] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 697fea6392524ef3995b71449704f594 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:06.815367 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.812122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.0.98.1:33525)
0929 14:42:06.812244 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.812248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.812339 (+ 91us) server_negotiation.cc:407] Connection header received
0929 14:42:06.812517 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.812521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.812575 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.812641 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:06.812926 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.813443 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.814296 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.814524 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.814612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.814899 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.814979 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.815036 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.815160 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.815225 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:06.815367 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.812017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33525 (local address 127.0.0.1:48636)
0929 14:42:06.812261 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.812276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.812395 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.812674 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.812676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.812685 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:06.812790 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.812796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.813554 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.813558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.814181 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.814189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.814636 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.814640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.814726 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.814772 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.815057 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.815072 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.815143 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.815236 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 14:42:06.818168 1947 raft_consensus.cc:1261] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Refusing update from remote peer 697fea6392524ef3995b71449704f594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.818598 1807 raft_consensus.cc:1261] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Refusing update from remote peer 697fea6392524ef3995b71449704f594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.819070 2027 consensus_queue.cc:1035] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.819260 2036 consensus_queue.cc:1035] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.820744 2023 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: TxnStatusTablet state changed. Reason: New leader 697fea6392524ef3995b71449704f594. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.820892 2023 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.820978 2022 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: TxnStatusTablet state changed. Reason: New leader 697fea6392524ef3995b71449704f594. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.821110 2022 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.823009 2027 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.823135 2027 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:06.823952 2022 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.824066 2022 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 46c05c4d274347b5abc18704a0dcbe14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.824700 2024 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.824801 2024 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 697fea6392524ef3995b71449704f594: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:06.826242 2023 tablet_replica.cc:439] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "697fea6392524ef3995b71449704f594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 } } peers { permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 } } peers { permanent_uuid: "46c05c4d274347b5abc18704a0dcbe14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34917 } } }
I20260929 14:42:06.826351 2023 tablet_replica.cc:442] T 1bf1b6c471584ed8a71529bffc01b9ba P 7c624311af5e40e48c1f0a4766e08b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:06.833693 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.828569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.0.98.3:34917)
0929 14:42:06.828815 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.828824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.829210 (+ 386us) server_negotiation.cc:407] Connection header received
0929 14:42:06.829348 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.829351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.829407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.829515 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:06.829871 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.830635 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.831569 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.831911 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.832009 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.832263 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.833003 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.833114 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.833195 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.833253 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260929 14:42:06.845568 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.841510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53443 (local address 127.0.98.1:33525)
0929 14:42:06.841640 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.841646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.841861 (+ 215us) server_negotiation.cc:407] Connection header received
0929 14:42:06.841969 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.841972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.842031 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.842117 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:06.842456 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.843260 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.844201 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.844557 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.844657 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.845055 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.845155 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.845230 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.845348 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.845411 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:42:06.845628 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.841392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33525 (local address 127.0.98.3:53443)
0929 14:42:06.841785 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.841798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.841891 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.842170 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.842173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.842180 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:06.842345 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.842350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.843392 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.843396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.844083 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.844093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.844712 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.844715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.844801 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.844921 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.845254 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.845261 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.845321 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.845405 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
I20260929 14:42:06.852578 1807 raft_consensus.cc:1261] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Refusing update from remote peer 46c05c4d274347b5abc18704a0dcbe14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.852845 1667 raft_consensus.cc:1261] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Refusing update from remote peer 46c05c4d274347b5abc18704a0dcbe14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:06.853529 2022 consensus_queue.cc:1035] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c624311af5e40e48c1f0a4766e08b35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.853801 2048 consensus_queue.cc:1035] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "697fea6392524ef3995b71449704f594" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:06.876543 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.872801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34917 (local address 127.0.98.1:48771)
0929 14:42:06.873195 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:06.873207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:06.873294 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:06.873566 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:06.873568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:06.873574 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:06.873695 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.873700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.874656 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.874666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.875385 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:06.875394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.875870 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.875873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:06.875950 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.875996 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.876270 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.876276 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:06.876332 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:42:06.876414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260929 14:42:06.876542 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.872925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48771 (local address 127.0.98.3:34917)
0929 14:42:06.873026 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.873031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.873272 (+ 241us) server_negotiation.cc:407] Connection header received
0929 14:42:06.873367 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.873370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.873420 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.873500 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:06.873782 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.874565 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.875488 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.875773 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.875861 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.876117 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:06.876200 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:06.876255 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.876347 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.876399 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 14:42:06.905912 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:06.906123 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:06.908756 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28550cc1e9f14a1abdcfeb314b8f2152: Sending DeleteTablet for 3 replicas of tablet 7aa9c69f8a214bcb84c72b84017bc013
I20260929 14:42:06.909713 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48630
I20260929 14:42:06.909729 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52070
I20260929 14:42:06.910084 2079 tablet_replica.cc:330] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: stopping tablet replica
I20260929 14:42:06.910211 2079 raft_consensus.cc:2229] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:06.910404 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aa9c69f8a214bcb84c72b84017bc013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39534
I20260929 14:42:06.910620 2079 raft_consensus.cc:2258] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.911386 2079 ts_tablet_manager.cc:1905] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.912132 2080 tablet_replica.cc:330] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: stopping tablet replica
I20260929 14:42:06.912400 2080 raft_consensus.cc:2229] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:06.912617 2080 raft_consensus.cc:2258] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.913584 2080 ts_tablet_manager.cc:1905] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.914624 2079 ts_tablet_manager.cc:1918] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:06.914710 2079 log.cc:1196] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/7aa9c69f8a214bcb84c72b84017bc013
I20260929 14:42:06.914968 2079 ts_tablet_manager.cc:1939] T 7aa9c69f8a214bcb84c72b84017bc013 P 46c05c4d274347b5abc18704a0dcbe14: Deleting consensus metadata
I20260929 14:42:06.915755 1497 catalog_manager.cc:4841] TS 46c05c4d274347b5abc18704a0dcbe14 (127.0.98.3:34917): tablet 7aa9c69f8a214bcb84c72b84017bc013 (table test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]) successfully deleted
I20260929 14:42:06.916080 2081 tablet_replica.cc:330] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: stopping tablet replica
I20260929 14:42:06.916214 2081 raft_consensus.cc:2229] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:06.916338 2081 raft_consensus.cc:2258] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:06.917050 2081 ts_tablet_manager.cc:1905] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:06.917838 2080 ts_tablet_manager.cc:1918] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:06.917955 2080 log.cc:1196] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/7aa9c69f8a214bcb84c72b84017bc013
I20260929 14:42:06.918227 2080 ts_tablet_manager.cc:1939] T 7aa9c69f8a214bcb84c72b84017bc013 P 697fea6392524ef3995b71449704f594: Deleting consensus metadata
I20260929 14:42:06.919297 1495 catalog_manager.cc:4841] TS 697fea6392524ef3995b71449704f594 (127.0.98.1:33525): tablet 7aa9c69f8a214bcb84c72b84017bc013 (table test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]) successfully deleted
I20260929 14:42:06.919970 2081 ts_tablet_manager.cc:1918] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:06.920053 2081 log.cc:1196] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/7aa9c69f8a214bcb84c72b84017bc013
I20260929 14:42:06.920351 2081 ts_tablet_manager.cc:1939] T 7aa9c69f8a214bcb84c72b84017bc013 P 7c624311af5e40e48c1f0a4766e08b35: Deleting consensus metadata
I20260929 14:42:06.921077 1495 catalog_manager.cc:4841] TS 7c624311af5e40e48c1f0a4766e08b35 (127.0.98.2:37623): tablet 7aa9c69f8a214bcb84c72b84017bc013 (table test-table [id=5a0cf1014ae94aa6b01e906e9fab7d5c]) successfully deleted
Sep 29 14:42:06 dist-test-slave-g8hp krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692925, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:06.933214 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.914074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.98.1:33525)
0929 14:42:06.914189 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.914195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.918681 (+ 4486us) server_negotiation.cc:407] Connection header received
0929 14:42:06.919819 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.919822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.919888 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.919936 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:06.921057 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.921839 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.923122 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.923452 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.923536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.928359 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:06.928378 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:06.928398 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:06.928436 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:06.930684 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.931300 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.931308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.931310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.931413 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.931836 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.931839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.931840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.932249 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:06.932562 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.932717 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.932854 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 14:42:06.933873 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48646
I20260929 14:42:06.945559 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:06.936286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.0.98.62:43611)
0929 14:42:06.936418 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:06.936422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:06.936609 (+ 187us) server_negotiation.cc:407] Connection header received
0929 14:42:06.936797 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:06.936800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:06.936849 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:06.936936 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:06.937876 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.938722 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.939899 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:06.940235 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:06.940346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:06.942069 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:06.942073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:06.942076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:06.942111 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:06.943748 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.944254 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.944257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.944258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.944349 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:06.944622 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:06.944624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:06.944625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:06.944838 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:06.945009 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:06.945248 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 14:42:06.945390 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 14:42:06.946112 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37804
I20260929 14:42:06.947613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1582
I20260929 14:42:06.956276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1722
I20260929 14:42:06.963503 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1862
I20260929 14:42:06.968823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 1478
2026-09-29T14:42:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4846 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:07 dist-test-slave-g8hp krb5kdc[2088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:07 dist-test-slave-g8hp krb5kdc[2088](info): set up 2 sockets
Sep 29 14:42:07 dist-test-slave-g8hp krb5kdc[2088](info): commencing operation
krb5kdc: starting...
W20260929 14:42:07.940725 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:08.065214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45701
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:08.180279 2101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:08.180601 2101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:08.180661 2101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:08.184975 2101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:08.185065 2101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:42:08.185088 2101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:08.185111 2101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:08.185132 2101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:08.185153 2101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:08.191269 2101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45701
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:08.192521 2101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:08.193655 2101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:08.202486 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:08.202478 2101 server_base.cc:1034] running on GCE node
W20260929 14:42:08.202471 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.202471 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:08.203480 2101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:08.204705 2101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:08.205888 2101 hybrid_clock.cc:648] HybridClock initialized: now 1790692928205864 us; error 40 us; skew 500 ppm
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:08.209404 2101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:08.210551 2101 webserver.cc:460] Webserver started at http://127.0.98.62:39833/ using document root <none> and password file <none>
I20260929 14:42:08.211174 2101 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:08.211265 2101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:08.211467 2101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:08.213678 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "17914b410ff64087b9f40de9ad7bad14"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "ed926eb71b8008cf0856b79c859a5920"
server_key_iv: "50e4b3a607ca378c7ac83afce9167c56"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.214241 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "17914b410ff64087b9f40de9ad7bad14"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "ed926eb71b8008cf0856b79c859a5920"
server_key_iv: "50e4b3a607ca378c7ac83afce9167c56"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.218202 2101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 14:42:08.221041 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.222508 2101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:08.222635 2101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "17914b410ff64087b9f40de9ad7bad14"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "ed926eb71b8008cf0856b79c859a5920"
server_key_iv: "50e4b3a607ca378c7ac83afce9167c56"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.222750 2101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:08.241444 2101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:08.244428 2101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:08.244597 2101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:08.252480 2101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45701
I20260929 14:42:08.252497 2178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45701 every 8 connection(s)
I20260929 14:42:08.253602 2101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:08.256629 2179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:08.262305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2101
I20260929 14:42:08.262418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:08.262660 392 external_mini_cluster.cc:1428] Setting key c7b8449d31aa22e5227c9db6afb0730a
I20260929 14:42:08.262820 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Bootstrap starting.
I20260929 14:42:08.265614 2179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:08.266537 2179 log.cc:824] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:08.268580 2179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: No bootstrap required, opened a new log
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:08.272609 2179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } }
I20260929 14:42:08.272915 2179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:08.272980 2179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17914b410ff64087b9f40de9ad7bad14, State: Initialized, Role: FOLLOWER
I20260929 14:42:08.273463 2179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } }
I20260929 14:42:08.273597 2179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:08.273660 2179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:08.273759 2179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:08.274715 2179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } }
I20260929 14:42:08.275100 2179 leader_election.cc:302] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17914b410ff64087b9f40de9ad7bad14; no voters:
I20260929 14:42:08.275463 2179 leader_election.cc:288] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:08.275684 2184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:08.275904 2184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [term 1 LEADER]: Becoming Leader. State: Replica: 17914b410ff64087b9f40de9ad7bad14, State: Running, Role: LEADER
I20260929 14:42:08.276232 2184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } }
I20260929 14:42:08.276813 2179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:08.277593 2186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17914b410ff64087b9f40de9ad7bad14. Latest consensus state: current_term: 1 leader_uuid: "17914b410ff64087b9f40de9ad7bad14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } } }
I20260929 14:42:08.278421 2186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:08.278461 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.264195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34672 (local address 127.0.98.62:45701)
0929 14:42:08.264778 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:08.264794 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:08.264833 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:42:08.265606 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:08.265627 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:08.266079 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:08.266484 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:08.267300 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.268436 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.269536 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.270034 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.270186 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.272940 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:08.272947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:08.272967 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:08.273006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:08.274774 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.275335 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.275341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.275346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.275487 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.275821 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.275824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.275826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.276404 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:08.276604 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:08.277011 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 14:42:08.277324 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":172,"threads_started":1}
I20260929 14:42:08.278299 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17914b410ff64087b9f40de9ad7bad14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17914b410ff64087b9f40de9ad7bad14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45701 } } }
I20260929 14:42:08.279731 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14 [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:08.282234 2199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:08.282316 2199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:08.282384 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:08.283170 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:08.289415 2198 catalog_manager.cc:1347] Generated new cluster ID: 7bfd1af770af467c96cf658f3c1e4dfb
I20260929 14:42:08.289515 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:08.326610 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:08.328022 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:08.336575 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Generated new TSK 0
I20260929 14:42:08.337361 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:08.373948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45701
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:08.487249 2205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:08.487548 2205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:08.487596 2205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:08.491892 2205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:08.491974 2205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:08.492024 2205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:42:08.492038 2205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:08.492079 2205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:08.498052 2205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:08.499174 2205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:08.500197 2205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:08.510517 2205 server_base.cc:1034] running on GCE node
W20260929 14:42:08.510519 2210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.510510 2213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.510746 2211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:08.511230 2205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:08.511899 2205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:08.513056 2205 hybrid_clock.cc:648] HybridClock initialized: now 1790692928513043 us; error 25 us; skew 500 ppm
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:08.515947 2205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:08.517014 2205 webserver.cc:460] Webserver started at http://127.0.98.1:38137/ using document root <none> and password file <none>
I20260929 14:42:08.517581 2205 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:08.517652 2205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:08.517864 2205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:08.520099 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "a8600003e1de4ecebf06b8f3af3f8178"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "123ac51a9acfab86eeb649b992960b38"
server_key_iv: "4dd2d440debd5a17bc3841310f2d3453"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.520669 2205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a8600003e1de4ecebf06b8f3af3f8178"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "123ac51a9acfab86eeb649b992960b38"
server_key_iv: "4dd2d440debd5a17bc3841310f2d3453"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.524814 2205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 14:42:08.527551 2220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.529008 2205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:08.529143 2205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "a8600003e1de4ecebf06b8f3af3f8178"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "123ac51a9acfab86eeb649b992960b38"
server_key_iv: "4dd2d440debd5a17bc3841310f2d3453"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.529254 2205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:08.545986 2205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:08.549425 2205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:08.549620 2205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:08.551214 2205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:08.551270 2205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.551328 2205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:08.551357 2205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.561755 2205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43965
I20260929 14:42:08.561952 2335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43965 every 8 connection(s)
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:08.562827 2205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:08.566891 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.552074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43385 (local address 127.0.98.62:45701)
0929 14:42:08.552190 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:08.552194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:08.553404 (+ 1210us) server_negotiation.cc:407] Connection header received
0929 14:42:08.554471 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:08.554475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:08.554535 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:08.554618 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:08.556328 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.557221 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.559228 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.559560 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.559679 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.562838 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:08.562842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:08.562845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:08.562880 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:08.564574 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.565190 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.565193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.565194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.565284 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.565678 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.565681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.565682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.565887 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:08.565963 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:08.566555 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 14:42:08.566709 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:42:08.567584 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.552679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.1:43385)
0929 14:42:08.553234 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:08.553281 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:08.554220 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:08.554854 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:08.554865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:08.555364 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:08.556135 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.556156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.557347 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.557351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.558999 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.559010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.559750 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.559755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.559994 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.560778 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:08.560797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:08.562624 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:08.564697 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.564702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.564715 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.565046 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.565390 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.565392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.565394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.565561 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.565997 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:08.566002 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:08.566260 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 14:42:08.566412 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 14:42:08.566680 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":167,"threads_started":1}
I20260929 14:42:08.568843 2336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45701
I20260929 14:42:08.569027 2336 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:08.569543 2336 heartbeater.cc:511] Master 127.0.98.62:45701 requested a full tablet report, sending...
I20260929 14:42:08.570519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2205
I20260929 14:42:08.570628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:08.570890 392 external_mini_cluster.cc:1428] Setting key 3810ef30b0e581acc49c6393b8bc2112
I20260929 14:42:08.571210 2131 ts_manager.cc:194] Registered new tserver with Master: a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965)
I20260929 14:42:08.572604 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43385
I20260929 14:42:08.579092 2342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.571204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.1:60113)
0929 14:42:08.571521 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:08.571535 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:08.571629 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:08.571850 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:08.571852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:08.572085 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:08.572208 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.572215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.573216 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.573219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.574104 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.574116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.574663 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.574667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.574758 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.575275 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:08.575287 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:08.575641 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:08.577599 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.577601 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.577603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.577826 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.578097 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.578099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.578100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.578170 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.578591 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:08.578593 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:08.578680 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 14:42:08.578773 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:42:08.578898 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260929 14:42:08.579141 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.571281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60113 (local address 127.0.98.62:45701)
0929 14:42:08.571394 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:08.571398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:08.571607 (+ 209us) server_negotiation.cc:407] Connection header received
0929 14:42:08.571698 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:08.571700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:08.571739 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:08.571815 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:08.572353 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.573098 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.574234 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.574537 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.574641 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.575784 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:08.575788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:08.575790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:08.575816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:08.577491 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.577906 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.577909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.577911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.578016 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.578246 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.578249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.578251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.578486 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:08.578582 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:08.578833 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 14:42:08.578975 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:08.601359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45701
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:08.714285 2345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:08.714633 2345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:08.714692 2345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:08.718916 2345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:08.718999 2345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:08.719043 2345 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:42:08.719060 2345 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:08.719113 2345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:08.724783 2345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:08.726035 2345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:08.727140 2345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:08.737581 2351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.737498 2350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.737495 2353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:08.737702 2345 server_base.cc:1034] running on GCE node
I20260929 14:42:08.738270 2345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:08.738963 2345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:08.740147 2345 hybrid_clock.cc:648] HybridClock initialized: now 1790692928740126 us; error 36 us; skew 500 ppm
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:08.743285 2345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:08.744342 2345 webserver.cc:460] Webserver started at http://127.0.98.2:45281/ using document root <none> and password file <none>
I20260929 14:42:08.744968 2345 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:08.745047 2345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:08.745258 2345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:08.747489 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "0862ae653c8e4cefb1d348e9e617b255"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "42c20e81d700314b60f5739e9d73037c"
server_key_iv: "b57f15dd0e3dae777db420e082101205"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.748054 2345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0862ae653c8e4cefb1d348e9e617b255"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "42c20e81d700314b60f5739e9d73037c"
server_key_iv: "b57f15dd0e3dae777db420e082101205"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.752743 2345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 14:42:08.755398 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.756668 2345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:42:08.756784 2345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "0862ae653c8e4cefb1d348e9e617b255"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "42c20e81d700314b60f5739e9d73037c"
server_key_iv: "b57f15dd0e3dae777db420e082101205"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.756948 2345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:08.767115 2345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:08.770306 2345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:08.770468 2345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:08.771852 2345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:08.771903 2345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.771943 2345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:08.771971 2345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:08.782121 2345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43393
I20260929 14:42:08.782158 2475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43393 every 8 connection(s)
I20260929 14:42:08.783224 2345 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:08.785974 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.772697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44927 (local address 127.0.98.62:45701)
0929 14:42:08.772814 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:08.772818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:08.773950 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 14:42:08.775076 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:08.775080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:08.775142 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:08.775243 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:08.776745 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.777485 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.778888 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.779246 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.779333 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.782227 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:08.782231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:08.782234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:08.782264 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:08.783757 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.784298 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.784301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.784303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.784402 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.784733 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.784736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.784738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.784992 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:08.785080 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:08.785667 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 14:42:08.785794 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:42:08.786811 2369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.773300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.2:44927)
0929 14:42:08.773799 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:08.773842 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:08.774836 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:08.775438 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:08.775448 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:08.775839 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:08.776539 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.776558 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.777632 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.777635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.778727 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.778738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.779463 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.779468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.779669 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.780273 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:08.780293 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:08.781993 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:08.783859 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.783865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.783881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.784207 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.784498 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.784501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.784503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.784637 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.785089 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:08.785096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:08.785357 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 14:42:08.785605 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 14:42:08.785927 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":145,"threads_started":1}
I20260929 14:42:08.787823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2345
I20260929 14:42:08.787935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:08.788162 2476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45701
I20260929 14:42:08.788239 392 external_mini_cluster.cc:1428] Setting key 68e824abfd2a1b614adf59b4b7592956
I20260929 14:42:08.788314 2476 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:08.789017 2476 heartbeater.cc:511] Master 127.0.98.62:45701 requested a full tablet report, sending...
I20260929 14:42:08.790086 2133 ts_manager.cc:194] Registered new tserver with Master: 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393)
I20260929 14:42:08.790853 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44927
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:08.799880 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.790416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38393 (local address 127.0.98.62:45701)
0929 14:42:08.792762 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:08.792766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:08.792778 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:08.792808 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:08.792810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:08.792844 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:08.792954 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:08.793435 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.794020 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.795183 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.795387 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.795457 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.796748 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:08.796751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:08.796753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:08.796775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:08.798159 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.798598 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.798601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.798602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.798692 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:08.798930 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:08.798933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:08.798934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:08.799185 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:08.799275 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:08.799621 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 14:42:08.799724 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2275}
I20260929 14:42:08.799893 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:08.790350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.2:38393)
0929 14:42:08.790605 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:08.790617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:08.790698 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:08.792986 (+ 2288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:08.792989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:08.793209 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:08.793330 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.793337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.794153 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.794156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.795059 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:08.795072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:08.795488 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:08.795491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:08.795583 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:08.796091 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:08.796105 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:08.796562 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:08.798258 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.798260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.798262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.798460 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.798776 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:08.798778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:08.798780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:08.798859 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:08.799288 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:08.799291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:08.799385 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 14:42:08.799553 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 14:42:08.799712 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:08.817358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45701
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:08.932844 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:08.933214 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:08.933290 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:08.937709 2485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:08.937799 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:08.937860 2485 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 14:42:08.937886 2485 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:08.937937 2485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:08.943634 2485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:08.944809 2485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:08.945922 2485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:08.956430 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.956430 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:08.956602 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:08.957077 2485 server_base.cc:1034] running on GCE node
I20260929 14:42:08.957669 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:08.958417 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:08.959692 2485 hybrid_clock.cc:648] HybridClock initialized: now 1790692928959653 us; error 53 us; skew 500 ppm
Sep 29 14:42:08 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:08.962953 2485 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:08.963977 2485 webserver.cc:460] Webserver started at http://127.0.98.3:45817/ using document root <none> and password file <none>
I20260929 14:42:08.964581 2485 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:08.964651 2485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:08.964854 2485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:08.967190 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "ec3228d75f41461c9cb24cf876823908"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "3fea8bb53fe20eb8aa79473dc0bafb59"
server_key_iv: "ef6a85ba6b7a903f5cad0968c9a34492"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.967805 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ec3228d75f41461c9cb24cf876823908"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "3fea8bb53fe20eb8aa79473dc0bafb59"
server_key_iv: "ef6a85ba6b7a903f5cad0968c9a34492"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.972043 2485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:08.975071 2500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.976661 2485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:08.976805 2485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "ec3228d75f41461c9cb24cf876823908"
format_stamp: "Formatted at 2026-09-29 14:42:08 on dist-test-slave-g8hp"
server_key: "3fea8bb53fe20eb8aa79473dc0bafb59"
server_key_iv: "ef6a85ba6b7a903f5cad0968c9a34492"
server_key_version: "encryptionkey@0"
I20260929 14:42:08.976989 2485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:08.994287 2485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:08.998023 2485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:08.998190 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:08.999776 2485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:08.999841 2485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:08.999897 2485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:08.999930 2485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:09 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:09.009982 2485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37353
I20260929 14:42:09.010924 2485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:09.011086 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37353 every 8 connection(s)
I20260929 14:42:09.014103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2485
I20260929 14:42:09.014225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:09.014415 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.000602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34653 (local address 127.0.98.62:45701)
0929 14:42:09.000718 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.000721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.001823 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 14:42:09.002880 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.002884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.002958 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.003018 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:09.004893 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.005707 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.007007 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.007293 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.007388 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.010599 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:09.010603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:09.010605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:09.010634 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:09.012252 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.012843 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.012846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.012847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.012983 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.013304 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.013308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.013310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.013509 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:09.013584 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.014126 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.014253 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:42:09.014539 392 external_mini_cluster.cc:1428] Setting key 15c0a19f15c8249280536d17ea90d173
I20260929 14:42:09.015409 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.001184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.3:34653)
0929 14:42:09.001673 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.001718 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.002625 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.003450 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.003459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.003991 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:09.004695 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.004716 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.005863 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.005867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.006858 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.006868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.007445 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.007449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.007634 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.008382 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:09.008407 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:09.010388 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:09.012399 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.012404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.012416 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.012709 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.013068 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.013071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.013072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.013209 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.013594 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:09.013599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:09.013859 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.014083 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.014465 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":135,"threads_started":1}
I20260929 14:42:09.016759 2616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45701
I20260929 14:42:09.016930 2616 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:09.017462 2616 heartbeater.cc:511] Master 127.0.98.62:45701 requested a full tablet report, sending...
I20260929 14:42:09.018451 2132 ts_manager.cc:194] Registered new tserver with Master: ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.019183 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34653
I20260929 14:42:09.025727 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.019050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44285 (local address 127.0.98.62:45701)
0929 14:42:09.019154 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.019158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.019169 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:09.019200 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.019202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.019236 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.019303 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:09.019789 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.020295 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.021180 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.021402 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.021488 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.022631 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:09.022634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:09.022636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:09.022658 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:09.024141 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.024580 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.024584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.024586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.024669 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.024945 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.024948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.024950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.025131 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:09.025222 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.025461 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.025567 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:42:09.025738 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.018733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.98.3:44285)
0929 14:42:09.019009 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.019023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.019103 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.019337 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.019340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.019568 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:09.019690 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.019697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.020410 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.020413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.021076 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.021084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.021497 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.021501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.021587 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.022115 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:09.022127 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:09.022500 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:09.024249 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.024252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.024254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.024487 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.024750 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.024753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.024755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.024832 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.025223 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:09.025226 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:09.025311 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.025395 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.025556 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
I20260929 14:42:09.028360 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:42:09 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:42:09 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:09.055349 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.045523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.62:45701)
0929 14:42:09.045643 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.045646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.045658 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:09.045701 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.045703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.045744 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.045830 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:09.046829 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.047388 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.048689 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.049031 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.049155 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.051661 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:09.051664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:09.051666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:09.051697 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:09.053166 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.053749 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.053752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.053754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.053912 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.054281 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.054284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.054286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.054606 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:09.054724 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.055058 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.055186 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:42:09.058899 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:09.062250 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:09.077692 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.071945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43965 (local address 127.0.0.1:51930)
0929 14:42:09.072497 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.072552 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.072675 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.073277 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.073283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.073308 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.073466 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.073474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.074552 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.074555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.076165 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.076174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.077325 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.077329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.077411 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.077434 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.077488 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.077543 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":29,"thread_start_us":102,"threads_started":1}
I20260929 14:42:09.077770 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.072111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.98.1:43965)
0929 14:42:09.072781 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.072788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.072813 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:42:09.072928 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.072935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.073131 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.073274 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.073587 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.074449 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.076313 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.077232 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.077329 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.077469 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.077525 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.077625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":62,"threads_started":1}
I20260929 14:42:09.077692 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.071674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.0.1:55048)
0929 14:42:09.073099 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.073116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.073203 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.073657 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.073660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.073683 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.073764 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.073770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.074934 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.074937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.076141 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.076150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.077038 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.077042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.077171 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.077208 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.077344 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.077514 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":65,"spinlock_wait_cycles":11648,"thread_start_us":80,"threads_started":1}
I20260929 14:42:09.078644 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.071846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.2:43393)
0929 14:42:09.073017 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.073029 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.073193 (+ 164us) server_negotiation.cc:407] Connection header received
0929 14:42:09.073311 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.073316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.073526 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.073693 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.073861 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.074840 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.076260 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.076932 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.077038 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.077238 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.078375 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.078494 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":110,"threads_started":1}
I20260929 14:42:09.079066 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.072799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.0.98.3:37353)
0929 14:42:09.073455 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.073466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.073689 (+ 223us) server_negotiation.cc:407] Connection header received
0929 14:42:09.073892 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.073897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.074078 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.074196 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.074626 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.075739 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.077264 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.078161 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.078239 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.078391 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.078786 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.078929 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":136,"threads_started":1}
I20260929 14:42:09.079412 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.072417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37353 (local address 127.0.0.1:49878)
0929 14:42:09.073597 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.073609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.073751 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.074358 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.074361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.074378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.074469 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.074475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.075865 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.075868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.077109 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.077121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.078487 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.078491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.078631 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.078654 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.079194 (+ 540us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.079275 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1120,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260929 14:42:09.080813 2270 tablet_service.cc:1467] Processing CreateTablet for tablet 912de05c43e64a52b92965320fc060c6 (DEFAULT_TABLE table=test-table [id=16dd234a57044bf2a7adf812b2c68fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.081331 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 912de05c43e64a52b92965320fc060c6 (DEFAULT_TABLE table=test-table [id=16dd234a57044bf2a7adf812b2c68fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.082029 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912de05c43e64a52b92965320fc060c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.082307 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912de05c43e64a52b92965320fc060c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.083724 2550 tablet_service.cc:1467] Processing CreateTablet for tablet 912de05c43e64a52b92965320fc060c6 (DEFAULT_TABLE table=test-table [id=16dd234a57044bf2a7adf812b2c68fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.084935 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 912de05c43e64a52b92965320fc060c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.089217 2639 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Bootstrap starting.
I20260929 14:42:09.091367 2640 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Bootstrap starting.
I20260929 14:42:09.091492 2639 tablet_bootstrap.cc:654] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.092226 2639 log.cc:824] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:09.093114 2642 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Bootstrap starting.
I20260929 14:42:09.094354 2639 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: No bootstrap required, opened a new log
I20260929 14:42:09.094569 2639 ts_tablet_manager.cc:1397] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:09.094658 2640 tablet_bootstrap.cc:654] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.095363 2640 log.cc:824] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:09.095448 2642 tablet_bootstrap.cc:654] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.096268 2642 log.cc:824] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:09.097360 2640 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: No bootstrap required, opened a new log
I20260929 14:42:09.097563 2640 ts_tablet_manager.cc:1397] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:09.098345 2642 tablet_bootstrap.cc:492] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: No bootstrap required, opened a new log
I20260929 14:42:09.098558 2642 ts_tablet_manager.cc:1397] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:09.098789 2639 raft_consensus.cc:348] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.099093 2639 raft_consensus.cc:374] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.099161 2639 raft_consensus.cc:729] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.099668 2639 consensus_queue.cc:260] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.100438 2336 heartbeater.cc:503] Master 127.0.98.62:45701 was elected leader, sending a full tablet report...
I20260929 14:42:09.100802 2639 ts_tablet_manager.cc:1428] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:42:09.102530 2642 raft_consensus.cc:348] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.102772 2642 raft_consensus.cc:374] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.102805 2642 raft_consensus.cc:729] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.103268 2642 consensus_queue.cc:260] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.103524 2640 raft_consensus.cc:348] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.103849 2640 raft_consensus.cc:374] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.103937 2640 raft_consensus.cc:729] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.104060 2616 heartbeater.cc:503] Master 127.0.98.62:45701 was elected leader, sending a full tablet report...
I20260929 14:42:09.104440 2642 ts_tablet_manager.cc:1428] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:42:09.104444 2640 consensus_queue.cc:260] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.105211 2476 heartbeater.cc:503] Master 127.0.98.62:45701 was elected leader, sending a full tablet report...
I20260929 14:42:09.105578 2640 ts_tablet_manager.cc:1428] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260929 14:42:09.261940 2617 tablet.cc:2369] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:09.265563 2645 raft_consensus.cc:482] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:09.265764 2645 raft_consensus.cc:504] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.266980 2645 leader_election.cc:288] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.272933 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.267817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37379 (local address 127.0.98.3:37353)
0929 14:42:09.267983 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.267987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.268159 (+ 172us) server_negotiation.cc:407] Connection header received
0929 14:42:09.268336 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.268339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.268389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.268544 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.268779 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.269574 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.271385 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.271998 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.272112 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.272178 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.272694 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.272773 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:09.272949 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.267378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57511 (local address 127.0.98.2:43393)
0929 14:42:09.267515 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.267520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.267701 (+ 181us) server_negotiation.cc:407] Connection header received
0929 14:42:09.267953 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.267956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.268011 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.268056 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.268635 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.269517 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.271385 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.272016 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.272112 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.272178 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.272694 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.272773 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:42:09.272969 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.267291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.98.1:57511)
0929 14:42:09.267609 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.267625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.267802 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.268343 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.268346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.268365 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.268510 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.268517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.269666 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.269670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.271225 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.271238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.272155 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.272159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.272535 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.272571 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.272663 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.272756 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":239,"thread_start_us":122,"threads_started":1}
I20260929 14:42:09.272970 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.267698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37353 (local address 127.0.98.1:37379)
0929 14:42:09.268082 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.268094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.268228 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.268560 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.268562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.268578 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.268679 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.268683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.269666 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.269670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.271234 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.271240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.272155 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.272159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.272531 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.272571 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.272655 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.272793 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":280,"thread_start_us":89,"threads_started":1}
I20260929 14:42:09.273793 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912de05c43e64a52b92965320fc060c6" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255" is_pre_election: true
I20260929 14:42:09.273793 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912de05c43e64a52b92965320fc060c6" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908" is_pre_election: true
I20260929 14:42:09.274144 2419 raft_consensus.cc:2454] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 0.
I20260929 14:42:09.274144 2570 raft_consensus.cc:2454] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 0.
I20260929 14:42:09.274679 2221 leader_election.cc:302] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8600003e1de4ecebf06b8f3af3f8178, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:09.274972 2645 raft_consensus.cc:2781] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:09.275070 2645 raft_consensus.cc:482] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:09.275120 2645 raft_consensus.cc:3037] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.276177 2645 raft_consensus.cc:504] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.276515 2645 leader_election.cc:288] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 election: Requested vote from peers 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.276907 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912de05c43e64a52b92965320fc060c6" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255"
I20260929 14:42:09.276974 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "912de05c43e64a52b92965320fc060c6" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908"
I20260929 14:42:09.277052 2419 raft_consensus.cc:3037] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.277127 2570 raft_consensus.cc:3037] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.278282 2419 raft_consensus.cc:2454] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 1.
I20260929 14:42:09.278484 2570 raft_consensus.cc:2454] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 1.
I20260929 14:42:09.278630 2221 leader_election.cc:302] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, a8600003e1de4ecebf06b8f3af3f8178; no voters:
I20260929 14:42:09.278805 2645 raft_consensus.cc:2781] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:09.279039 2645 raft_consensus.cc:686] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 LEADER]: Becoming Leader. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Running, Role: LEADER
I20260929 14:42:09.279366 2645 consensus_queue.cc:237] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } }
I20260929 14:42:09.282980 2132 catalog_manager.cc:5495] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8600003e1de4ecebf06b8f3af3f8178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } health_report { overall_health: HEALTHY } } }
W20260929 14:42:09.284693 2477 tablet.cc:2369] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:09.314531 2337 tablet.cc:2369] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 14:42:09 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:09.343031 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.334715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45701 (local address 127.0.0.1:34696)
0929 14:42:09.334998 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.335011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.335122 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.335341 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.335343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.335769 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:09.335864 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.335871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.336585 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.336588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.337200 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.337209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.337637 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.337640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.337713 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.338272 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:09.338284 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:09.339820 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:09.341472 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.341483 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.341490 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.341734 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.341984 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:09.341987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:09.341989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:09.342061 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:09.342423 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:09.342432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:09.342567 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.342664 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.342834 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
I20260929 14:42:09.343031 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.334777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.0.98.62:45701)
0929 14:42:09.334938 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.334942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.335080 (+ 138us) server_negotiation.cc:407] Connection header received
0929 14:42:09.335203 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.335206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.335244 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.335306 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:09.335983 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.336489 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.337308 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.337532 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.337617 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.340012 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:09.340015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:09.340017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:09.340040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:09.341366 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.341831 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.341834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.341835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.341907 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:09.342135 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:09.342138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:09.342140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:09.342322 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:09.342407 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.342725 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.342917 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":40}
I20260929 14:42:09.347548 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34696:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:42:09.348289 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:09.353370 2410 tablet_service.cc:1467] Processing CreateTablet for tablet c740eabbf9e344a18bd879a542e6a347 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=872188ae03e1417b84165f408824406b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:09.353539 2550 tablet_service.cc:1467] Processing CreateTablet for tablet c740eabbf9e344a18bd879a542e6a347 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=872188ae03e1417b84165f408824406b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:09.353375 2270 tablet_service.cc:1467] Processing CreateTablet for tablet c740eabbf9e344a18bd879a542e6a347 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=872188ae03e1417b84165f408824406b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:09.354135 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c740eabbf9e344a18bd879a542e6a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.354135 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c740eabbf9e344a18bd879a542e6a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.354137 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c740eabbf9e344a18bd879a542e6a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.356628 2640 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: Bootstrap starting.
I20260929 14:42:09.357010 2642 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: Bootstrap starting.
I20260929 14:42:09.357220 2639 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: Bootstrap starting.
I20260929 14:42:09.357617 2640 tablet_bootstrap.cc:654] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.357981 2642 tablet_bootstrap.cc:654] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.358351 2639 tablet_bootstrap.cc:654] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.359284 2640 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: No bootstrap required, opened a new log
I20260929 14:42:09.359380 2640 ts_tablet_manager.cc:1397] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:09.359488 2639 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: No bootstrap required, opened a new log
I20260929 14:42:09.359571 2639 ts_tablet_manager.cc:1397] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:42:09.359838 2642 tablet_bootstrap.cc:492] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: No bootstrap required, opened a new log
I20260929 14:42:09.359934 2642 ts_tablet_manager.cc:1397] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:09.360056 2639 raft_consensus.cc:348] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360179 2639 raft_consensus.cc:374] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.360217 2639 raft_consensus.cc:729] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.360183 2640 raft_consensus.cc:348] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360294 2640 raft_consensus.cc:374] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.360327 2640 raft_consensus.cc:729] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.360347 2639 consensus_queue.cc:260] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360354 2642 raft_consensus.cc:348] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360445 2642 raft_consensus.cc:374] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.360476 2642 raft_consensus.cc:729] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.360463 2640 consensus_queue.cc:260] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360581 2642 consensus_queue.cc:260] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.360617 2645 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.360723 2640 ts_tablet_manager.cc:1428] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:42:09.360770 2645 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.360841 2646 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.360999 2646 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.360827 2642 ts_tablet_manager.cc:1428] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:42:09.361322 2647 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.361479 2647 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.362162 2639 ts_tablet_manager.cc:1428] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:09.568141 2647 raft_consensus.cc:482] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:09.568392 2647 raft_consensus.cc:504] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.569667 2647 leader_election.cc:288] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.574123 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.570365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45679 (local address 127.0.98.3:37353)
0929 14:42:09.570479 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.570483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.570496 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:09.570609 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.570611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.570654 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.570730 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.570996 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.571514 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.572983 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.573548 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.573661 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.573728 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.573905 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.574006 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:42:09.574131 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.569933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37353 (local address 127.0.98.2:45679)
0929 14:42:09.570379 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.570395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.570496 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.570740 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.570742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.570755 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.570888 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.570893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.571619 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.571623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.572780 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.572790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.573686 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.573689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.573781 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.573802 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.573873 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.573953 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260929 14:42:09.574265 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.570006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56137 (local address 127.0.98.1:43965)
0929 14:42:09.570124 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.570128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.570392 (+ 264us) server_negotiation.cc:407] Connection header received
0929 14:42:09.570606 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.570610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.570656 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.570700 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:09.571109 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.571867 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.573160 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.573796 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.573887 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.573944 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.574096 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.574165 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:42:09.574290 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.569902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43965 (local address 127.0.98.2:56137)
0929 14:42:09.570302 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.570315 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.570469 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.570843 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.570845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.570859 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:09.570990 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.570996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.571977 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.571981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.573057 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.573064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.573899 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.573903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.573988 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.574008 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.574071 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.574129 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":21,"thread_start_us":82,"threads_started":1}
I20260929 14:42:09.574983 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c740eabbf9e344a18bd879a542e6a347" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908" is_pre_election: true
I20260929 14:42:09.575099 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c740eabbf9e344a18bd879a542e6a347" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178" is_pre_election: true
I20260929 14:42:09.575165 2570 raft_consensus.cc:2454] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 0.
I20260929 14:42:09.575320 2290 raft_consensus.cc:2454] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 0.
I20260929 14:42:09.575762 2364 leader_election.cc:302] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, a8600003e1de4ecebf06b8f3af3f8178, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:09.576030 2647 raft_consensus.cc:2781] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:09.576089 2647 raft_consensus.cc:482] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:09.576131 2647 raft_consensus.cc:3037] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.577183 2647 raft_consensus.cc:504] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.577603 2647 leader_election.cc:288] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 election: Requested vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.577984 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c740eabbf9e344a18bd879a542e6a347" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178"
I20260929 14:42:09.577984 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c740eabbf9e344a18bd879a542e6a347" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908"
I20260929 14:42:09.578140 2570 raft_consensus.cc:3037] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.578153 2290 raft_consensus.cc:3037] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.579296 2290 raft_consensus.cc:2454] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 1.
I20260929 14:42:09.579290 2570 raft_consensus.cc:2454] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 1.
I20260929 14:42:09.579607 2364 leader_election.cc:302] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, a8600003e1de4ecebf06b8f3af3f8178; no voters:
I20260929 14:42:09.579836 2647 raft_consensus.cc:2781] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:09.580029 2647 raft_consensus.cc:686] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 LEADER]: Becoming Leader. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Running, Role: LEADER
I20260929 14:42:09.580410 2647 consensus_queue.cc:237] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.582468 2663 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: TxnStatusTablet state changed. Reason: New leader 0862ae653c8e4cefb1d348e9e617b255. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.582574 2663 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:09.582835 2665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:42:09.583053 2665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:42:09.583915 2132 catalog_manager.cc:5495] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:09.605031 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.601270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.0.1:55060)
0929 14:42:09.601453 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.601476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.601579 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.601963 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.601966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.601976 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:09.602076 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.602081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.602792 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.602795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.603426 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.603435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.603871 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.603874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.603960 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.604005 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.604690 (+ 685us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.604706 (+ 16us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.604787 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.604914 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:42:09.605031 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.601558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.2:43393)
0929 14:42:09.601704 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.601708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.601720 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:09.601790 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.601793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.601846 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.601935 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:09.602174 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.602681 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.603542 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.603765 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.603852 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.604140 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.604598 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.604699 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.604799 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.604855 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:42:09.610479 2290 raft_consensus.cc:1261] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Refusing update from remote peer 0862ae653c8e4cefb1d348e9e617b255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.610505 2570 raft_consensus.cc:1261] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Refusing update from remote peer 0862ae653c8e4cefb1d348e9e617b255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.611137 2663 consensus_queue.cc:1035] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:09.611394 2647 consensus_queue.cc:1035] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:09.614660 2645 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: TxnStatusTablet state changed. Reason: New leader 0862ae653c8e4cefb1d348e9e617b255. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.614794 2645 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.614945 2646 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: TxnStatusTablet state changed. Reason: New leader 0862ae653c8e4cefb1d348e9e617b255. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.615067 2646 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.618592 2669 mvcc.cc:205] Tried to move back new op lower bound from 7334678239676833792 to 7334678239564320768. Current Snapshot: MvccSnapshot[applied={T|T < 7334678239676833792}]
I20260929 14:42:09.618834 2645 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.618954 2645 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P a8600003e1de4ecebf06b8f3af3f8178: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.619342 2646 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.619463 2646 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P ec3228d75f41461c9cb24cf876823908: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:09.621531 2647 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.621660 2647 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:09.622069 2663 tablet_replica.cc:439] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } } }
I20260929 14:42:09.622186 2663 tablet_replica.cc:442] T c740eabbf9e344a18bd879a542e6a347 P 0862ae653c8e4cefb1d348e9e617b255: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:09.623842 2670 mvcc.cc:205] Tried to move back new op lower bound from 7334678239676833792 to 7334678239564320768. Current Snapshot: MvccSnapshot[applied={T|T < 7334678239676833792}]
I20260929 14:42:09.633453 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.629508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.0.98.1:43965)
0929 14:42:09.629631 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.629636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.629812 (+ 176us) server_negotiation.cc:407] Connection header received
0929 14:42:09.630035 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.630038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.630095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.630199 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:09.630434 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.630944 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.631816 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.632074 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.632167 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.632464 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.633027 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.633144 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.633232 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.633295 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:42:09.645622 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.642237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49383 (local address 127.0.98.2:43393)
0929 14:42:09.642386 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.642390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.642493 (+ 103us) server_negotiation.cc:407] Connection header received
0929 14:42:09.642641 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.642644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.642690 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.642751 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:09.643044 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.643614 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.644507 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.644722 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.644810 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.645166 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.645248 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.645306 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.645411 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.645467 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 14:42:09.645635 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.642106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.98.1:49383)
0929 14:42:09.642411 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.642427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.642526 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.642774 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.642776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.642784 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:09.642914 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.642919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.643727 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.643730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.644400 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.644412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.644828 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.644832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.644950 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.645001 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.645321 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.645327 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.645400 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.645486 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260929 14:42:09.653380 2570 raft_consensus.cc:1261] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Refusing update from remote peer a8600003e1de4ecebf06b8f3af3f8178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.653685 2419 raft_consensus.cc:1261] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Refusing update from remote peer a8600003e1de4ecebf06b8f3af3f8178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.653921 2650 consensus_queue.cc:1035] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:09.654309 2650 consensus_queue.cc:1035] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:09.661360 2668 mvcc.cc:205] Tried to move back new op lower bound from 7334678239855415296 to 7334678238331260928. Current Snapshot: MvccSnapshot[applied={T|T < 7334678239855415296}]
I20260929 14:42:09.678699 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.674721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34789 (local address 127.0.98.1:43965)
0929 14:42:09.674847 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:09.674851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:09.675071 (+ 220us) server_negotiation.cc:407] Connection header received
0929 14:42:09.675253 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:09.675256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:09.675321 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:09.675402 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:09.675664 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.676452 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.677411 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.677770 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.677873 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.678171 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.678286 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.678347 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:09.678486 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 14:42:09.678555 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:42:09.678742 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:09.674634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43965 (local address 127.0.98.2:34789)
0929 14:42:09.674989 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:09.675001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:09.675105 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:09.675434 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:09.675437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:09.675446 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:09.675558 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.675564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.676566 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.676571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.677298 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:09.677309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:09.677875 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:09.677878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:09.677952 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:09.678040 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:09.678387 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:09.678394 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:42:09.678455 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 14:42:09.678550 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":122,"threads_started":1}
I20260929 14:42:09.708847 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:09.709139 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:09.713390 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Sending DeleteTablet for 3 replicas of tablet 912de05c43e64a52b92965320fc060c6
I20260929 14:42:09.714617 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 912de05c43e64a52b92965320fc060c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 14:42:09.715250 2691 tablet_replica.cc:330] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: stopping tablet replica
I20260929 14:42:09.715165 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet 912de05c43e64a52b92965320fc060c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260929 14:42:09.715648 2691 raft_consensus.cc:2229] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:09.715857 2691 raft_consensus.cc:2258] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:09.716039 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet 912de05c43e64a52b92965320fc060c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49878
I20260929 14:42:09.717686 2692 tablet_replica.cc:330] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: stopping tablet replica
I20260929 14:42:09.717633 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:09.718168 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:09.718367 2692 raft_consensus.cc:2229] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:09.718925 2692 raft_consensus.cc:2258] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:09.720577 2692 ts_tablet_manager.cc:1905] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:09.724644 2410 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd (DEFAULT_TABLE table=test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.724699 2550 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd (DEFAULT_TABLE table=test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.725061 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2e8d3382e5466eb728f45b0083e7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.725145 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2e8d3382e5466eb728f45b0083e7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.727684 2640 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Bootstrap starting.
I20260929 14:42:09.728639 2691 ts_tablet_manager.cc:1905] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:09.729063 2693 tablet_replica.cc:330] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: stopping tablet replica
I20260929 14:42:09.729247 2640 tablet_bootstrap.cc:654] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.729419 2693 raft_consensus.cc:2229] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:09.729653 2693 raft_consensus.cc:2258] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:09.730767 2642 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Bootstrap starting.
I20260929 14:42:09.731217 2693 ts_tablet_manager.cc:1905] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:09.731813 2642 tablet_bootstrap.cc:654] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.730615 2270 tablet_service.cc:1467] Processing CreateTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd (DEFAULT_TABLE table=test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:09.733350 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2e8d3382e5466eb728f45b0083e7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:09.733606 2691 ts_tablet_manager.cc:1918] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:09.733709 2691 log.cc:1196] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/912de05c43e64a52b92965320fc060c6
I20260929 14:42:09.734035 2691 ts_tablet_manager.cc:1939] T 912de05c43e64a52b92965320fc060c6 P 0862ae653c8e4cefb1d348e9e617b255: Deleting consensus metadata
I20260929 14:42:09.734094 2693 ts_tablet_manager.cc:1918] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:09.734169 2693 log.cc:1196] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/912de05c43e64a52b92965320fc060c6
I20260929 14:42:09.734440 2693 ts_tablet_manager.cc:1939] T 912de05c43e64a52b92965320fc060c6 P ec3228d75f41461c9cb24cf876823908: Deleting consensus metadata
I20260929 14:42:09.735371 2117 catalog_manager.cc:4841] TS 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393): tablet 912de05c43e64a52b92965320fc060c6 (table test-table [id=16dd234a57044bf2a7adf812b2c68fed]) successfully deleted
I20260929 14:42:09.735579 2117 catalog_manager.cc:4841] TS ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353): tablet 912de05c43e64a52b92965320fc060c6 (table test-table [id=16dd234a57044bf2a7adf812b2c68fed]) successfully deleted
I20260929 14:42:09.735930 2639 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Bootstrap starting.
I20260929 14:42:09.724191 2692 ts_tablet_manager.cc:1918] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:09.736582 2692 log.cc:1196] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/912de05c43e64a52b92965320fc060c6
I20260929 14:42:09.736991 2639 tablet_bootstrap.cc:654] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:09.737133 2692 ts_tablet_manager.cc:1939] T 912de05c43e64a52b92965320fc060c6 P a8600003e1de4ecebf06b8f3af3f8178: Deleting consensus metadata
I20260929 14:42:09.738442 2120 catalog_manager.cc:4841] TS a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965): tablet 912de05c43e64a52b92965320fc060c6 (table test-table [id=16dd234a57044bf2a7adf812b2c68fed]) successfully deleted
I20260929 14:42:09.739193 2639 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: No bootstrap required, opened a new log
I20260929 14:42:09.739285 2639 ts_tablet_manager.cc:1397] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 14:42:09.739434 2640 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: No bootstrap required, opened a new log
I20260929 14:42:09.739524 2640 ts_tablet_manager.cc:1397] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260929 14:42:09.739748 2639 raft_consensus.cc:348] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.739886 2639 raft_consensus.cc:374] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.739919 2639 raft_consensus.cc:729] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.739970 2640 raft_consensus.cc:348] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.740089 2640 raft_consensus.cc:374] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.740049 2639 consensus_queue.cc:260] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.740118 2640 raft_consensus.cc:729] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.740231 2640 consensus_queue.cc:260] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.740301 2639 ts_tablet_manager.cc:1428] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:09.740456 2640 ts_tablet_manager.cc:1428] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:09.744107 2642 tablet_bootstrap.cc:492] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: No bootstrap required, opened a new log
I20260929 14:42:09.744258 2642 ts_tablet_manager.cc:1397] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260929 14:42:09.744894 2642 raft_consensus.cc:348] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.745064 2642 raft_consensus.cc:374] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:09.745103 2642 raft_consensus.cc:729] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Initialized, Role: FOLLOWER
I20260929 14:42:09.745239 2642 consensus_queue.cc:260] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.745563 2642 ts_tablet_manager.cc:1428] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 14:42:09.814769 2645 raft_consensus.cc:482] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:09.814913 2645 raft_consensus.cc:504] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.815303 2645 leader_election.cc:288] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.815716 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e8d3382e5466eb728f45b0083e7cd" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255" is_pre_election: true
I20260929 14:42:09.815771 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e8d3382e5466eb728f45b0083e7cd" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908" is_pre_election: true
I20260929 14:42:09.815889 2570 raft_consensus.cc:2454] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 0.
I20260929 14:42:09.815917 2419 raft_consensus.cc:2454] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 0.
I20260929 14:42:09.816186 2221 leader_election.cc:302] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, a8600003e1de4ecebf06b8f3af3f8178; no voters:
I20260929 14:42:09.816401 2645 raft_consensus.cc:2781] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:09.816470 2645 raft_consensus.cc:482] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:09.816514 2645 raft_consensus.cc:3037] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.817600 2645 raft_consensus.cc:504] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.817946 2645 leader_election.cc:288] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 election: Requested vote from peers 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:09.818331 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e8d3382e5466eb728f45b0083e7cd" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908"
I20260929 14:42:09.818347 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2e8d3382e5466eb728f45b0083e7cd" candidate_uuid: "a8600003e1de4ecebf06b8f3af3f8178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255"
I20260929 14:42:09.818449 2570 raft_consensus.cc:3037] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.818478 2419 raft_consensus.cc:3037] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:09.819401 2570 raft_consensus.cc:2454] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 1.
I20260929 14:42:09.819664 2419 raft_consensus.cc:2454] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8600003e1de4ecebf06b8f3af3f8178 in term 1.
I20260929 14:42:09.819715 2221 leader_election.cc:302] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8600003e1de4ecebf06b8f3af3f8178, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:09.819868 2645 raft_consensus.cc:2781] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:09.819950 2645 raft_consensus.cc:686] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 1 LEADER]: Becoming Leader. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Running, Role: LEADER
I20260929 14:42:09.820088 2645 consensus_queue.cc:237] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:09.822063 2131 catalog_manager.cc:5495] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a8600003e1de4ecebf06b8f3af3f8178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:09.849764 2570 raft_consensus.cc:1261] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Refusing update from remote peer a8600003e1de4ecebf06b8f3af3f8178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.849892 2419 raft_consensus.cc:1261] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Refusing update from remote peer a8600003e1de4ecebf06b8f3af3f8178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:09.850185 2645 consensus_queue.cc:1035] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:09.850345 2684 consensus_queue.cc:1035] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:42:09.866956 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:42:11.421423 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:11.421635 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:11.424295 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Sending DeleteTablet for 3 replicas of tablet ab2e8d3382e5466eb728f45b0083e7cd
I20260929 14:42:11.424953 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260929 14:42:11.425006 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 14:42:11.425117 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2e8d3382e5466eb728f45b0083e7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49878
I20260929 14:42:11.425331 2721 tablet_replica.cc:330] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: stopping tablet replica
I20260929 14:42:11.425463 2721 raft_consensus.cc:2229] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:11.425485 2722 tablet_replica.cc:330] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: stopping tablet replica
I20260929 14:42:11.425582 2722 raft_consensus.cc:2229] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:11.425599 2721 raft_consensus.cc:2258] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:11.425776 2722 raft_consensus.cc:2258] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:11.426137 2723 tablet_replica.cc:330] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: stopping tablet replica
I20260929 14:42:11.426262 2723 raft_consensus.cc:2229] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:11.426271 2721 ts_tablet_manager.cc:1905] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.426384 2723 raft_consensus.cc:2258] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:11.426940 2722 ts_tablet_manager.cc:1905] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.429061 2721 ts_tablet_manager.cc:1918] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:42:11.429143 2721 log.cc:1196] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/ab2e8d3382e5466eb728f45b0083e7cd
I20260929 14:42:11.429379 2721 ts_tablet_manager.cc:1939] T ab2e8d3382e5466eb728f45b0083e7cd P 0862ae653c8e4cefb1d348e9e617b255: Deleting consensus metadata
I20260929 14:42:11.429441 2723 ts_tablet_manager.cc:1905] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.429535 2722 ts_tablet_manager.cc:1918] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:42:11.429606 2722 log.cc:1196] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/ab2e8d3382e5466eb728f45b0083e7cd
I20260929 14:42:11.429821 2722 ts_tablet_manager.cc:1939] T ab2e8d3382e5466eb728f45b0083e7cd P a8600003e1de4ecebf06b8f3af3f8178: Deleting consensus metadata
I20260929 14:42:11.430832 2117 catalog_manager.cc:4841] TS 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393): tablet ab2e8d3382e5466eb728f45b0083e7cd (table test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]) successfully deleted
I20260929 14:42:11.430907 2120 catalog_manager.cc:4841] TS a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965): tablet ab2e8d3382e5466eb728f45b0083e7cd (table test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]) successfully deleted
I20260929 14:42:11.431804 2723 ts_tablet_manager.cc:1918] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:42:11.431867 2723 log.cc:1196] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/ab2e8d3382e5466eb728f45b0083e7cd
I20260929 14:42:11.432058 2723 ts_tablet_manager.cc:1939] T ab2e8d3382e5466eb728f45b0083e7cd P ec3228d75f41461c9cb24cf876823908: Deleting consensus metadata
I20260929 14:42:11.433233 2117 catalog_manager.cc:4841] TS ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353): tablet ab2e8d3382e5466eb728f45b0083e7cd (table test-table [id=3d0553c85392453e85f4e7b4aca3f0b1]) successfully deleted
Sep 29 14:42:11 dist-test-slave-g8hp krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:42:11 dist-test-slave-g8hp krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:11.461462 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.445362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.98.62:45701)
0929 14:42:11.445661 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.445669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.445765 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:42:11.445911 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.445915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.445968 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.446047 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:11.446770 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.447337 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.448196 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.448431 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.448519 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.458220 (+ 9701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:11.458224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:11.458226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:11.458260 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:11.459776 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:11.460234 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:11.460237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:11.460239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:11.460317 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:11.460562 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:11.460566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:11.460567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:11.460765 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:11.460901 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.461164 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.461292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260929 14:42:11.464167 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:11.464632 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:11.469784 2270 tablet_service.cc:1467] Processing CreateTablet for tablet dcc064515b3b463e8ef6248423ee37f0 (DEFAULT_TABLE table=test-table [id=9383ff6656f54696adf68ffd70559be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.469965 2410 tablet_service.cc:1467] Processing CreateTablet for tablet dcc064515b3b463e8ef6248423ee37f0 (DEFAULT_TABLE table=test-table [id=9383ff6656f54696adf68ffd70559be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.469964 2550 tablet_service.cc:1467] Processing CreateTablet for tablet dcc064515b3b463e8ef6248423ee37f0 (DEFAULT_TABLE table=test-table [id=9383ff6656f54696adf68ffd70559be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.470155 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc064515b3b463e8ef6248423ee37f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.470247 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc064515b3b463e8ef6248423ee37f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.470247 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc064515b3b463e8ef6248423ee37f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.472481 2732 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Bootstrap starting.
I20260929 14:42:11.472935 2734 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Bootstrap starting.
I20260929 14:42:11.473354 2732 tablet_bootstrap.cc:654] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.473697 2734 tablet_bootstrap.cc:654] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.474221 2733 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Bootstrap starting.
I20260929 14:42:11.474442 2732 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: No bootstrap required, opened a new log
I20260929 14:42:11.474524 2732 ts_tablet_manager.cc:1397] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 14:42:11.474753 2734 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: No bootstrap required, opened a new log
I20260929 14:42:11.474849 2734 ts_tablet_manager.cc:1397] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:11.474890 2732 raft_consensus.cc:348] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.475003 2732 raft_consensus.cc:374] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.475057 2732 raft_consensus.cc:729] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.475198 2732 consensus_queue.cc:260] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.475452 2733 tablet_bootstrap.cc:654] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.475534 2732 ts_tablet_manager.cc:1428] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:11.475682 2734 raft_consensus.cc:348] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.475819 2734 raft_consensus.cc:374] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.475875 2734 raft_consensus.cc:729] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.476033 2734 consensus_queue.cc:260] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.476449 2734 ts_tablet_manager.cc:1428] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:11.477499 2733 tablet_bootstrap.cc:492] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: No bootstrap required, opened a new log
I20260929 14:42:11.477596 2733 ts_tablet_manager.cc:1397] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:11.478302 2733 raft_consensus.cc:348] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.478432 2733 raft_consensus.cc:374] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.478494 2733 raft_consensus.cc:729] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.478639 2733 consensus_queue.cc:260] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.479048 2733 ts_tablet_manager.cc:1428] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:11.638485 2738 raft_consensus.cc:482] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:11.638689 2738 raft_consensus.cc:504] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.639767 2738 leader_election.cc:288] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393)
I20260929 14:42:11.645421 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.640292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36345 (local address 127.0.98.1:43965)
0929 14:42:11.640540 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.640548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.640561 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:11.640646 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.640649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.640699 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.640798 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:11.641282 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.642229 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.643999 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.644650 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.644767 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.644836 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.645192 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.645266 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260929 14:42:11.645421 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.640115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57173 (local address 127.0.98.2:43393)
0929 14:42:11.640409 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.640417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.640495 (+ 78us) server_negotiation.cc:407] Connection header received
0929 14:42:11.640652 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.640654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.640699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.640802 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:11.641282 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.642131 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.643999 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.644672 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.644766 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.644836 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.645176 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.645266 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":113,"threads_started":1}
I20260929 14:42:11.645474 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.640023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.98.3:57173)
0929 14:42:11.640410 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:11.640426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:11.640552 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:11.640827 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:11.640830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:11.640898 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:11.641131 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.641139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.642273 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.642278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.643843 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.643857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.644800 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.644805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.645007 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.645054 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 14:42:11.645151 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:42:11.645274 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":230,"spinlock_wait_cycles":99968,"thread_start_us":175,"threads_started":1}
I20260929 14:42:11.645481 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.640023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43965 (local address 127.0.98.3:36345)
0929 14:42:11.640471 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:11.640482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:11.640558 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:11.640827 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:11.640830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:11.640898 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:11.641131 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.641139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.642354 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.642357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.643847 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.643857 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.644800 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.644805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.644993 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.645052 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 14:42:11.645144 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:42:11.645271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":221,"spinlock_wait_cycles":8448,"thread_start_us":94,"threads_started":1}
I20260929 14:42:11.645985 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc064515b3b463e8ef6248423ee37f0" candidate_uuid: "ec3228d75f41461c9cb24cf876823908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255" is_pre_election: true
I20260929 14:42:11.646157 2419 raft_consensus.cc:2454] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3228d75f41461c9cb24cf876823908 in term 0.
I20260929 14:42:11.646346 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc064515b3b463e8ef6248423ee37f0" candidate_uuid: "ec3228d75f41461c9cb24cf876823908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178" is_pre_election: true
I20260929 14:42:11.646503 2290 raft_consensus.cc:2454] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3228d75f41461c9cb24cf876823908 in term 0.
I20260929 14:42:11.646566 2501 leader_election.cc:302] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:11.646839 2738 raft_consensus.cc:2781] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:11.646906 2738 raft_consensus.cc:482] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:11.646936 2738 raft_consensus.cc:3037] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.648065 2738 raft_consensus.cc:504] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.648439 2738 leader_election.cc:288] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [CANDIDATE]: Term 1 election: Requested vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393)
I20260929 14:42:11.648811 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc064515b3b463e8ef6248423ee37f0" candidate_uuid: "ec3228d75f41461c9cb24cf876823908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178"
I20260929 14:42:11.648903 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc064515b3b463e8ef6248423ee37f0" candidate_uuid: "ec3228d75f41461c9cb24cf876823908" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0862ae653c8e4cefb1d348e9e617b255"
I20260929 14:42:11.648963 2290 raft_consensus.cc:3037] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.649037 2419 raft_consensus.cc:3037] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.649904 2290 raft_consensus.cc:2454] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3228d75f41461c9cb24cf876823908 in term 1.
I20260929 14:42:11.649938 2419 raft_consensus.cc:2454] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3228d75f41461c9cb24cf876823908 in term 1.
I20260929 14:42:11.650223 2504 leader_election.cc:302] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8600003e1de4ecebf06b8f3af3f8178, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:11.650408 2738 raft_consensus.cc:2781] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:11.650630 2738 raft_consensus.cc:686] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 1 LEADER]: Becoming Leader. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Running, Role: LEADER
I20260929 14:42:11.650877 2738 consensus_queue.cc:237] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.653157 2133 catalog_manager.cc:5495] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3228d75f41461c9cb24cf876823908 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ec3228d75f41461c9cb24cf876823908" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } health_report { overall_health: HEALTHY } } }
I20260929 14:42:11.675412 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.671140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.0.98.3:37353)
0929 14:42:11.671414 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.671422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.671435 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:11.671632 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.671636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.671707 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.671838 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:11.672273 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.672851 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.673884 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.674139 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.674226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.674507 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.675014 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.675119 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.675190 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.675249 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":112,"threads_started":1}
I20260929 14:42:11.686625 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.683342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41843 (local address 127.0.98.2:43393)
0929 14:42:11.683453 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.683456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.683490 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:42:11.683620 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.683622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.683670 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.683731 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:11.683995 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.684500 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.685506 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.685777 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.685857 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.686171 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.686259 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.686321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.686448 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.686501 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:42:11.686659 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.683104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.98.3:41843)
0929 14:42:11.683424 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:11.683440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:11.683543 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:11.683792 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:11.683795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:11.683803 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:11.683908 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.683913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.684622 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.684625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.685370 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.685381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.685883 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.685885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.685961 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.686008 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.686361 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.686368 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:42:11.686432 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:42:11.686501 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260929 14:42:11.692462 2290 raft_consensus.cc:1261] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Refusing update from remote peer ec3228d75f41461c9cb24cf876823908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:11.692452 2419 raft_consensus.cc:1261] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Refusing update from remote peer ec3228d75f41461c9cb24cf876823908: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:11.692960 2743 consensus_queue.cc:1035] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:11.693140 2738 consensus_queue.cc:1035] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:11.710353 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.706749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37353 (local address 127.0.98.2:35681)
0929 14:42:11.706976 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:11.706989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:11.707067 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:11.707391 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:11.707395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:11.707403 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:11.707502 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.707507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.708228 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.708231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.708849 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.708892 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.709396 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.709400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.709479 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.709567 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.710004 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.710010 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:11.710085 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 14:42:11.710185 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260929 14:42:11.710469 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.706799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35681 (local address 127.0.98.3:37353)
0929 14:42:11.707028 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.707033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.707046 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:11.707259 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.707262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.707313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.707378 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:11.707598 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.708103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.708991 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.709258 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.709361 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.709784 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.709880 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.709951 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.710271 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.710341 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:42:11.733376 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:11.733577 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:11.736126 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Sending DeleteTablet for 3 replicas of tablet dcc064515b3b463e8ef6248423ee37f0
I20260929 14:42:11.736765 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc064515b3b463e8ef6248423ee37f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260929 14:42:11.736999 2722 tablet_replica.cc:330] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: stopping tablet replica
I20260929 14:42:11.737082 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc064515b3b463e8ef6248423ee37f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49878
I20260929 14:42:11.737120 2722 raft_consensus.cc:2229] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:11.737258 2722 raft_consensus.cc:2258] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:11.737296 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc064515b3b463e8ef6248423ee37f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 14:42:11.737433 2723 tablet_replica.cc:330] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: stopping tablet replica
I20260929 14:42:11.737470 2721 tablet_replica.cc:330] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: stopping tablet replica
I20260929 14:42:11.737593 2721 raft_consensus.cc:2229] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:11.737596 2723 raft_consensus.cc:2229] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:11.737707 2721 raft_consensus.cc:2258] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:11.737640 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:42:11.737955 2723 raft_consensus.cc:2258] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:42:11.738132 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:11.738528 2721 ts_tablet_manager.cc:1905] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.739251 2723 ts_tablet_manager.cc:1905] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.741211 2721 ts_tablet_manager.cc:1918] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:11.741281 2721 log.cc:1196] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/dcc064515b3b463e8ef6248423ee37f0
I20260929 14:42:11.741514 2721 ts_tablet_manager.cc:1939] T dcc064515b3b463e8ef6248423ee37f0 P 0862ae653c8e4cefb1d348e9e617b255: Deleting consensus metadata
I20260929 14:42:11.741850 2723 ts_tablet_manager.cc:1918] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:11.741914 2723 log.cc:1196] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/dcc064515b3b463e8ef6248423ee37f0
I20260929 14:42:11.742136 2723 ts_tablet_manager.cc:1939] T dcc064515b3b463e8ef6248423ee37f0 P ec3228d75f41461c9cb24cf876823908: Deleting consensus metadata
I20260929 14:42:11.743057 2117 catalog_manager.cc:4841] TS ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353): tablet dcc064515b3b463e8ef6248423ee37f0 (table test-table [id=9383ff6656f54696adf68ffd70559be8]) successfully deleted
I20260929 14:42:11.743844 2550 tablet_service.cc:1467] Processing CreateTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 (DEFAULT_TABLE table=test-table [id=565850aedb504b5d9935c5ce6ff6ba0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.743855 2270 tablet_service.cc:1467] Processing CreateTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 (DEFAULT_TABLE table=test-table [id=565850aedb504b5d9935c5ce6ff6ba0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.744118 2550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38e408d35fc4bfcb45f3b134f2c5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.744118 2270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38e408d35fc4bfcb45f3b134f2c5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.746513 2733 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Bootstrap starting.
I20260929 14:42:11.746873 2117 catalog_manager.cc:4841] TS 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393): tablet dcc064515b3b463e8ef6248423ee37f0 (table test-table [id=9383ff6656f54696adf68ffd70559be8]) successfully deleted
I20260929 14:42:11.747109 2410 tablet_service.cc:1467] Processing CreateTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 (DEFAULT_TABLE table=test-table [id=565850aedb504b5d9935c5ce6ff6ba0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:11.747370 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38e408d35fc4bfcb45f3b134f2c5b16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:11.747617 2733 tablet_bootstrap.cc:654] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.748287 2732 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: Bootstrap starting.
I20260929 14:42:11.749143 2732 tablet_bootstrap.cc:654] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.749390 2733 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: No bootstrap required, opened a new log
I20260929 14:42:11.749466 2733 ts_tablet_manager.cc:1397] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:11.749648 2734 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Bootstrap starting.
I20260929 14:42:11.749903 2733 raft_consensus.cc:348] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.750044 2733 raft_consensus.cc:374] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.750078 2733 raft_consensus.cc:729] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3228d75f41461c9cb24cf876823908, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.750200 2733 consensus_queue.cc:260] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.750299 2732 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: No bootstrap required, opened a new log
I20260929 14:42:11.750373 2732 ts_tablet_manager.cc:1397] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:11.750496 2734 tablet_bootstrap.cc:654] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:11.750753 2732 raft_consensus.cc:348] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.750855 2732 raft_consensus.cc:374] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.750885 2732 raft_consensus.cc:729] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8600003e1de4ecebf06b8f3af3f8178, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.750995 2732 consensus_queue.cc:260] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.751250 2732 ts_tablet_manager.cc:1428] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:11.751871 2733 ts_tablet_manager.cc:1428] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:11.752836 2722 ts_tablet_manager.cc:1905] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:11.753194 2734 tablet_bootstrap.cc:492] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: No bootstrap required, opened a new log
I20260929 14:42:11.753269 2734 ts_tablet_manager.cc:1397] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 14:42:11.753618 2734 raft_consensus.cc:348] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.753715 2734 raft_consensus.cc:374] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:11.753742 2734 raft_consensus.cc:729] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Initialized, Role: FOLLOWER
I20260929 14:42:11.753866 2734 consensus_queue.cc:260] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.754117 2734 ts_tablet_manager.cc:1428] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:11.755445 2722 ts_tablet_manager.cc:1918] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:11.755514 2722 log.cc:1196] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/dcc064515b3b463e8ef6248423ee37f0
I20260929 14:42:11.755724 2722 ts_tablet_manager.cc:1939] T dcc064515b3b463e8ef6248423ee37f0 P a8600003e1de4ecebf06b8f3af3f8178: Deleting consensus metadata
I20260929 14:42:11.756716 2120 catalog_manager.cc:4841] TS a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965): tablet dcc064515b3b463e8ef6248423ee37f0 (table test-table [id=9383ff6656f54696adf68ffd70559be8]) successfully deleted
I20260929 14:42:11.759542 2663 raft_consensus.cc:482] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:11.759722 2663 raft_consensus.cc:504] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.760174 2663 leader_election.cc:288] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:11.760703 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38e408d35fc4bfcb45f3b134f2c5b16" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178" is_pre_election: true
I20260929 14:42:11.760772 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38e408d35fc4bfcb45f3b134f2c5b16" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908" is_pre_election: true
I20260929 14:42:11.760929 2570 raft_consensus.cc:2454] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 0.
I20260929 14:42:11.761158 2290 raft_consensus.cc:2454] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 0.
I20260929 14:42:11.761288 2361 leader_election.cc:302] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:11.761458 2663 raft_consensus.cc:2781] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:11.761509 2663 raft_consensus.cc:482] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:11.761543 2663 raft_consensus.cc:3037] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.762784 2663 raft_consensus.cc:504] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.763137 2663 leader_election.cc:288] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 election: Requested vote from peers a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965), ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353)
I20260929 14:42:11.763453 2290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38e408d35fc4bfcb45f3b134f2c5b16" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8600003e1de4ecebf06b8f3af3f8178"
I20260929 14:42:11.763450 2570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38e408d35fc4bfcb45f3b134f2c5b16" candidate_uuid: "0862ae653c8e4cefb1d348e9e617b255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3228d75f41461c9cb24cf876823908"
I20260929 14:42:11.763558 2570 raft_consensus.cc:3037] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.763561 2290 raft_consensus.cc:3037] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:11.764410 2570 raft_consensus.cc:2454] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 1.
I20260929 14:42:11.764663 2361 leader_election.cc:302] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0862ae653c8e4cefb1d348e9e617b255, ec3228d75f41461c9cb24cf876823908; no voters:
I20260929 14:42:11.764669 2290 raft_consensus.cc:2454] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0862ae653c8e4cefb1d348e9e617b255 in term 1.
I20260929 14:42:11.764814 2663 raft_consensus.cc:2781] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:11.764945 2663 raft_consensus.cc:686] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 LEADER]: Becoming Leader. State: Replica: 0862ae653c8e4cefb1d348e9e617b255, State: Running, Role: LEADER
I20260929 14:42:11.765070 2663 consensus_queue.cc:237] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } }
I20260929 14:42:11.766767 2133 catalog_manager.cc:5495] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0862ae653c8e4cefb1d348e9e617b255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0862ae653c8e4cefb1d348e9e617b255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:11.784719 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.780989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.98.2:43393)
0929 14:42:11.781114 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.781119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.781174 (+ 55us) server_negotiation.cc:407] Connection header received
0929 14:42:11.781313 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.781316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.781369 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.781432 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:11.781634 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.782207 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.783398 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.783725 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.783811 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.784128 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.784217 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.784268 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.784546 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.784594 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:42:11.794531 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.791007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60571 (local address 127.0.98.2:43393)
0929 14:42:11.791092 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:11.791096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:11.791117 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:42:11.791159 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:11.791162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:11.791206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:11.791273 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:11.791473 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.792247 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.793408 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.793742 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.793830 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.794082 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.794166 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.794221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:11.794299 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 14:42:11.794370 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":29}
I20260929 14:42:11.794556 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:11.790758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43393 (local address 127.0.98.2:60571)
0929 14:42:11.790924 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:11.790939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:11.791019 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:11.791288 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:11.791291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:11.791297 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:11.791387 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.791393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.792342 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.792345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.793285 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:11.793296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:11.793838 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:11.793842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:11.793949 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:11.793981 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:11.794231 (+ 250us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:11.794236 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:42:11.794292 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:42:11.794403 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":9}
I20260929 14:42:11.800848 2570 raft_consensus.cc:1261] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Refusing update from remote peer 0862ae653c8e4cefb1d348e9e617b255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:11.800992 2290 raft_consensus.cc:1261] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178 [term 1 FOLLOWER]: Refusing update from remote peer 0862ae653c8e4cefb1d348e9e617b255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:11.801252 2702 consensus_queue.cc:1035] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3228d75f41461c9cb24cf876823908" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:11.801378 2707 consensus_queue.cc:1035] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8600003e1de4ecebf06b8f3af3f8178" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:42:11.819191 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 14:42:13.393893 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:13.394058 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:13.396644 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17914b410ff64087b9f40de9ad7bad14: Sending DeleteTablet for 3 replicas of tablet a38e408d35fc4bfcb45f3b134f2c5b16
I20260929 14:42:13.397277 2270 tablet_service.cc:1514] Processing DeleteTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260929 14:42:13.397324 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55048
I20260929 14:42:13.397502 2550 tablet_service.cc:1514] Processing DeleteTablet for tablet a38e408d35fc4bfcb45f3b134f2c5b16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49878
I20260929 14:42:13.397673 2783 tablet_replica.cc:330] T a38e408d35fc4bfcb45f3b134f2c5b16 P a8600003e1de4ecebf06b8f3af3f8178: stopping tablet replica
I20260929 14:42:13.397678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2205
I20260929 14:42:13.398409 2784 tablet_replica.cc:330] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: stopping tablet replica
I20260929 14:42:13.398561 2784 raft_consensus.cc:2229] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:13.398872 2784 raft_consensus.cc:2258] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:13.399649 2784 ts_tablet_manager.cc:1905] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:13.399859 2785 tablet_replica.cc:330] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: stopping tablet replica
I20260929 14:42:13.400048 2785 raft_consensus.cc:2229] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:13.400209 2785 raft_consensus.cc:2258] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:13.400918 2785 ts_tablet_manager.cc:1905] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:13.401849 2784 ts_tablet_manager.cc:1918] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:42:13.401923 2784 log.cc:1196] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/a38e408d35fc4bfcb45f3b134f2c5b16
I20260929 14:42:13.402187 2784 ts_tablet_manager.cc:1939] T a38e408d35fc4bfcb45f3b134f2c5b16 P 0862ae653c8e4cefb1d348e9e617b255: Deleting consensus metadata
I20260929 14:42:13.403199 2117 catalog_manager.cc:4841] TS 0862ae653c8e4cefb1d348e9e617b255 (127.0.98.2:43393): tablet a38e408d35fc4bfcb45f3b134f2c5b16 (table test-table [id=565850aedb504b5d9935c5ce6ff6ba0c]) successfully deleted
I20260929 14:42:13.403503 2785 ts_tablet_manager.cc:1918] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 14:42:13.403613 2785 log.cc:1196] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/a38e408d35fc4bfcb45f3b134f2c5b16
I20260929 14:42:13.403856 2785 ts_tablet_manager.cc:1939] T a38e408d35fc4bfcb45f3b134f2c5b16 P ec3228d75f41461c9cb24cf876823908: Deleting consensus metadata
I20260929 14:42:13.404511 2117 catalog_manager.cc:4841] TS ec3228d75f41461c9cb24cf876823908 (127.0.98.3:37353): tablet a38e408d35fc4bfcb45f3b134f2c5b16 (table test-table [id=565850aedb504b5d9935c5ce6ff6ba0c]) successfully deleted
W20260929 14:42:13.405345 2120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43965 (error 108)
I20260929 14:42:13.405593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2345
I20260929 14:42:13.407035 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:13.406620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43965 (local address 127.0.0.1:58616)
0929 14:42:13.406840 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:13.406916 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
W20260929 14:42:13.407155 2120 catalog_manager.cc:4584] TS a8600003e1de4ecebf06b8f3af3f8178 (127.0.98.1:43965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a38e408d35fc4bfcb45f3b134f2c5b16: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43965: connect: Connection refused (error 111)
I20260929 14:42:13.413177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2485
I20260929 14:42:13.418856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2101
2026-09-29T14:42:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6449 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:13 dist-test-slave-g8hp krb5kdc[2790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:13 dist-test-slave-g8hp krb5kdc[2790](info): set up 2 sockets
Sep 29 14:42:13 dist-test-slave-g8hp krb5kdc[2790](info): commencing operation
krb5kdc: starting...
W20260929 14:42:14.380656 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:14 dist-test-slave-g8hp krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:14.505033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44193
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:14.617118 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:14.617414 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:14.617455 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:14.621908 2803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:14.621974 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:14.621994 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:14.622010 2803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:14.622027 2803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:14.627557 2803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:14.628721 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:14.629875 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:14.638512 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:14.638520 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:14.638567 2803 server_base.cc:1034] running on GCE node
W20260929 14:42:14.638520 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:14.639307 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:14.640450 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:14.641636 2803 hybrid_clock.cc:648] HybridClock initialized: now 1790692934641507 us; error 143 us; skew 500 ppm
Sep 29 14:42:14 dist-test-slave-g8hp krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:14.644696 2803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:14.645833 2803 webserver.cc:460] Webserver started at http://127.0.98.62:44419/ using document root <none> and password file <none>
I20260929 14:42:14.646380 2803 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:14.646461 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:14.646672 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:14.648761 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "068a4d67121440778b0a7c1ebab7da11"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "1f043839d77ff418121cbd09f5de17eb"
server_key_iv: "04a9a43e4073639e6d083fb92cd9c93c"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.649355 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "068a4d67121440778b0a7c1ebab7da11"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "1f043839d77ff418121cbd09f5de17eb"
server_key_iv: "04a9a43e4073639e6d083fb92cd9c93c"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.653246 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:14.655810 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:14.657291 2803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:14.657419 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "068a4d67121440778b0a7c1ebab7da11"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "1f043839d77ff418121cbd09f5de17eb"
server_key_iv: "04a9a43e4073639e6d083fb92cd9c93c"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.657526 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:14.678385 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:14.681856 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:14.682053 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:14.689056 2803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44193
I20260929 14:42:14.689078 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44193 every 8 connection(s)
I20260929 14:42:14.690073 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:14.691550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2803
I20260929 14:42:14.691658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:14.691931 392 external_mini_cluster.cc:1428] Setting key 352e1213fd55de3238369723dff43dc1
I20260929 14:42:14.693501 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:14.699203 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11: Bootstrap starting.
Sep 29 14:42:14 dist-test-slave-g8hp krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:14.701877 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:14.702632 2871 log.cc:824] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:14.705219 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11: No bootstrap required, opened a new log
I20260929 14:42:14.706578 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:14.693437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.62:44193)
0929 14:42:14.693881 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:14.693949 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:14.693989 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:42:14.694633 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:14.694650 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:14.695004 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:14.695316 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:14.696097 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.697130 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.697973 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.698456 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.698602 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:14.701448 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:14.701454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:14.701470 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:14.701505 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:14.703275 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:14.703811 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:14.703817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:14.703822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:14.703952 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:14.704396 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:14.704400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:14.704402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:14.704836 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:14.705096 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:14.705441 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 14:42:14.705761 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":145,"threads_started":1}
I20260929 14:42:14.709410 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260929 14:42:14.709657 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:14.709717 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068a4d67121440778b0a7c1ebab7da11, State: Initialized, Role: FOLLOWER
I20260929 14:42:14.710139 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260929 14:42:14.710268 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:14.710310 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:14.710408 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:14.711373 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260929 14:42:14.711716 2871 leader_election.cc:302] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 068a4d67121440778b0a7c1ebab7da11; no voters:
I20260929 14:42:14.712034 2871 leader_election.cc:288] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:14.712172 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:14.712396 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [term 1 LEADER]: Becoming Leader. State: Replica: 068a4d67121440778b0a7c1ebab7da11, State: Running, Role: LEADER
I20260929 14:42:14.712781 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } }
I20260929 14:42:14.713276 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:14.714514 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 068a4d67121440778b0a7c1ebab7da11. Latest consensus state: current_term: 1 leader_uuid: "068a4d67121440778b0a7c1ebab7da11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } } }
I20260929 14:42:14.714578 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "068a4d67121440778b0a7c1ebab7da11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "068a4d67121440778b0a7c1ebab7da11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44193 } } }
I20260929 14:42:14.714684 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:14.714637 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:14.715224 2886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:14.718039 2886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:14.724114 2886 catalog_manager.cc:1347] Generated new cluster ID: 3bb080058b3049c49a42abcf96d35988
I20260929 14:42:14.724190 2886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:14.731572 2886 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:14.732576 2886 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:14.739437 2886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 068a4d67121440778b0a7c1ebab7da11: Generated new TSK 0
I20260929 14:42:14.740209 2886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:14.780277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:14.893661 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:14.893959 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:14.894003 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:14.898190 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:14.898262 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:14.898322 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:14.903678 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:14.904965 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:14.906052 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:14.915983 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:14.916109 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:14.916011 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:14.916201 2897 server_base.cc:1034] running on GCE node
I20260929 14:42:14.916760 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:14.917505 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:14.918681 2897 hybrid_clock.cc:648] HybridClock initialized: now 1790692934918647 us; error 47 us; skew 500 ppm
Sep 29 14:42:14 dist-test-slave-g8hp krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:14.921550 2897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:14.922555 2897 webserver.cc:460] Webserver started at http://127.0.98.1:34521/ using document root <none> and password file <none>
I20260929 14:42:14.923130 2897 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:14.923202 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:14.923408 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:14.925585 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "b45d9e13139344cb8b09f6a6b80fdbe1"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "53cc60cf639a3c843d99e0fb3b468460"
server_key_iv: "a1500bb40536cb11b86b4e07d06bd5cc"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.926146 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b45d9e13139344cb8b09f6a6b80fdbe1"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "53cc60cf639a3c843d99e0fb3b468460"
server_key_iv: "a1500bb40536cb11b86b4e07d06bd5cc"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.929992 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:42:14.932416 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:14.933931 2897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:14.934060 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "b45d9e13139344cb8b09f6a6b80fdbe1"
format_stamp: "Formatted at 2026-09-29 14:42:14 on dist-test-slave-g8hp"
server_key: "53cc60cf639a3c843d99e0fb3b468460"
server_key_iv: "a1500bb40536cb11b86b4e07d06bd5cc"
server_key_version: "encryptionkey@0"
I20260929 14:42:14.934172 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:14.944222 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:14.947817 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:14.948079 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:14.948719 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:14.949688 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:14.949754 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:14.949812 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:14.949837 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:14.958565 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35407
I20260929 14:42:14.958587 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35407 every 8 connection(s)
I20260929 14:42:14.959542 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:14.967334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 2897
I20260929 14:42:14.967449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:14.967782 392 external_mini_cluster.cc:1428] Setting key 79e64ae549b016ae17b3cad1116cae4a
Sep 29 14:42:14 dist-test-slave-g8hp krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:14.974155 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:14.961399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36869 (local address 127.0.98.62:44193)
0929 14:42:14.961529 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:14.961533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:14.962413 (+ 880us) server_negotiation.cc:407] Connection header received
0929 14:42:14.963893 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:14.963896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:14.963956 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:14.964029 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:14.965324 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.965863 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.967027 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.967268 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.967358 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:14.970257 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:14.970262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:14.970272 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:14.970303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:14.971695 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:14.972321 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:14.972324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:14.972325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:14.972407 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:14.972841 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:14.972844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:14.972845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:14.973089 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:14.973175 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:14.973903 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 14:42:14.974028 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:14.975380 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:14.961733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.1:36869)
0929 14:42:14.962250 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:14.962296 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:14.963652 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:14.964222 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:14.964229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:14.964606 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:14.965165 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:14.965181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.965988 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.965991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:14.966897 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:14.966906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:14.967427 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:14.967431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:14.967615 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:14.968222 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:14.968243 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:14.970072 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:14.971815 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:14.971821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:14.971840 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:14.972190 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:14.972489 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:14.972492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:14.972494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:14.972678 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:14.973197 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:14.973204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:14.973588 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 14:42:14.973815 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:42:14.974215 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":123,"threads_started":1}
I20260929 14:42:14.976846 3026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260929 14:42:14.977111 3026 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:14.977643 3026 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
I20260929 14:42:14.979247 2835 ts_manager.cc:194] Registered new tserver with Master: b45d9e13139344cb8b09f6a6b80fdbe1 (127.0.98.1:35407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:14.980641 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:14.996956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:15.109457 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:15.109784 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:15.109830 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:15.114080 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:15.114163 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:15.114252 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:15.119937 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:15.121069 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:15.122102 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:15.131946 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:15.132043 3031 server_base.cc:1034] running on GCE node
W20260929 14:42:15.131946 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:15.132251 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:15.132681 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:15.133409 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:15.134589 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790692935134567 us; error 37 us; skew 500 ppm
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:15.137522 3031 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:15.138499 3031 webserver.cc:460] Webserver started at http://127.0.98.2:39733/ using document root <none> and password file <none>
I20260929 14:42:15.139040 3031 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:15.139106 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:15.139300 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:15.141407 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a3b248bb1814be094f82ea63c350fde"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "962641e069b3b124fb1ef6881c79fff7"
server_key_iv: "8e320579e022ec4b010df2ed8ea7e82f"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.141950 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a3b248bb1814be094f82ea63c350fde"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "962641e069b3b124fb1ef6881c79fff7"
server_key_iv: "8e320579e022ec4b010df2ed8ea7e82f"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.145643 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:15.148063 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.149468 3031 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:15.149581 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "8a3b248bb1814be094f82ea63c350fde"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "962641e069b3b124fb1ef6881c79fff7"
server_key_iv: "8e320579e022ec4b010df2ed8ea7e82f"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.149696 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:15.161201 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:15.164100 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:15.164279 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:15.164851 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:15.165699 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:15.165746 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.165794 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:15.165814 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.174170 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36043
I20260929 14:42:15.174216 3159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36043 every 8 connection(s)
I20260929 14:42:15.175140 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:15.183264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3031
I20260929 14:42:15.183368 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:15.183614 392 external_mini_cluster.cc:1428] Setting key bc0c6bca43999b0ed134dca23653d5dd
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:15.189683 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:15.176961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34631 (local address 127.0.98.62:44193)
0929 14:42:15.177085 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:15.177088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:15.177977 (+ 889us) server_negotiation.cc:407] Connection header received
0929 14:42:15.179364 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:15.179368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:15.179425 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:15.179495 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:15.180719 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.181319 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.182556 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.182779 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.182855 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:15.185680 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:15.185683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:15.185685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:15.185716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:15.187428 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:15.187964 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:15.187968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:15.187969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:15.188058 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:15.188433 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:15.188436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:15.188437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:15.188657 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:15.188760 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:15.189365 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 14:42:15.189496 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:42:15.190528 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:15.177377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.2:34631)
0929 14:42:15.177826 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:15.177869 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:15.179149 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:15.179683 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:15.179691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:15.180069 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:15.180571 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:15.180585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.181455 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.181459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:15.182441 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:15.182455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.182934 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.182938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:15.183086 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:15.183697 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:15.183719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:15.185480 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:15.187561 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:15.187567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:15.187582 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:15.187858 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:15.188184 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:15.188189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:15.188190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:15.188342 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:15.188782 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:15.188787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:15.189113 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 14:42:15.189293 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 14:42:15.189572 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260929 14:42:15.191771 3160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260929 14:42:15.192054 3160 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:15.192679 3160 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:15.193953 2835 ts_manager.cc:194] Registered new tserver with Master: 8a3b248bb1814be094f82ea63c350fde (127.0.98.2:36043)
I20260929 14:42:15.194564 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:15.212018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:15.324625 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:15.324936 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:15.324994 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:15.329167 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:15.329228 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:15.329288 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:15.335117 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:15.336182 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:15.337179 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:15.347124 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:15.347195 3165 server_base.cc:1034] running on GCE node
W20260929 14:42:15.347007 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:15.346980 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:15.347905 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:15.348559 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:15.349721 3165 hybrid_clock.cc:648] HybridClock initialized: now 1790692935349675 us; error 62 us; skew 500 ppm
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:15.352643 3165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:15.353677 3165 webserver.cc:460] Webserver started at http://127.0.98.3:44111/ using document root <none> and password file <none>
I20260929 14:42:15.354235 3165 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:15.354296 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:15.354488 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:15.356537 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "a00f04088bc44120bbe44f171a104d69"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "8b08cd1b531104fcc6d436fce2a87ca8"
server_key_iv: "14d2deb4c6be27c4b3c157d38af55221"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.357101 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a00f04088bc44120bbe44f171a104d69"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "8b08cd1b531104fcc6d436fce2a87ca8"
server_key_iv: "14d2deb4c6be27c4b3c157d38af55221"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.361171 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:42:15.363806 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.365231 3165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:15.365352 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "a00f04088bc44120bbe44f171a104d69"
format_stamp: "Formatted at 2026-09-29 14:42:15 on dist-test-slave-g8hp"
server_key: "8b08cd1b531104fcc6d436fce2a87ca8"
server_key_iv: "14d2deb4c6be27c4b3c157d38af55221"
server_key_version: "encryptionkey@0"
I20260929 14:42:15.365458 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:15.383483 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:15.387087 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:15.387284 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:15.387907 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:15.388741 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:15.388789 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.388852 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:15.388921 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:15.397708 3165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32849
I20260929 14:42:15.397761 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32849 every 8 connection(s)
I20260929 14:42:15.398635 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:15.408965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3165
I20260929 14:42:15.409082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:15.409385 392 external_mini_cluster.cc:1428] Setting key a122e731793b2ed6ecfe1cd6c8825682
I20260929 14:42:15.413002 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:15.400592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45031 (local address 127.0.98.62:44193)
0929 14:42:15.400755 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:15.400759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:15.401744 (+ 985us) server_negotiation.cc:407] Connection header received
0929 14:42:15.403144 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:15.403147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:15.403197 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:15.403275 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:15.404602 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.405236 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.406513 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.406723 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.406798 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:15.409459 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:15.409463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:15.409466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:15.409496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:15.411039 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:15.411539 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:15.411543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:15.411545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:15.411637 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:15.411952 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:15.411954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:15.411955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:15.412134 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:15.412207 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:15.412723 (+ 516us) server_negotiation.cc:299] Negotiation successful
0929 14:42:15.412836 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:42:15.414093 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:15.401067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44193 (local address 127.0.98.3:45031)
0929 14:42:15.401565 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:15.401612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:15.402921 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:15.403491 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:15.403498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:15.403922 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:15.404465 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:15.404480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.405372 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.405375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:15.406399 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:15.406409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:15.406866 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:15.406869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:15.407030 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:15.407576 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:15.407592 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:15.409271 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:15.411149 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:15.411161 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:15.411176 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:15.411448 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:15.411730 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:15.411734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:15.411735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:15.411877 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:15.412246 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:15.412253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:15.412521 (+ 268us) client_negotiation.cc:769] Sending connection context
0929 14:42:15.412695 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:42:15.413092 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260929 14:42:15.415211 3294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44193
I20260929 14:42:15.415462 3294 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:15.416000 3294 heartbeater.cc:511] Master 127.0.98.62:44193 requested a full tablet report, sending...
I20260929 14:42:15.417069 2835 ts_manager.cc:194] Registered new tserver with Master: a00f04088bc44120bbe44f171a104d69 (127.0.98.3:32849)
I20260929 14:42:15.417629 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45031
I20260929 14:42:15.423494 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:42:15.435719 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:15.434390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.0.98.62:44193)
0929 14:42:15.434504 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:15.434508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:15.434654 (+ 146us) server_negotiation.cc:407] Connection header received
0929 14:42:15.434775 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:15.434777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:15.434827 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:15.434896 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:15.435591 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34196: BlockingRecv error: recv got EOF from 127.0.0.1:34196 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 14:42:15.436759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2897
I20260929 14:42:15.441731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3031
I20260929 14:42:15.446535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3165
I20260929 14:42:15.451247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 2803
2026-09-29T14:42:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2030 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[3306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[3306](info): set up 2 sockets
Sep 29 14:42:15 dist-test-slave-g8hp krb5kdc[3306](info): commencing operation
krb5kdc: starting...
W20260929 14:42:16.416672 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:16.542689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:16.655169 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:16.655462 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:16.655503 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:16.659672 3319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:16.659751 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:16.659770 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:16.659786 3319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:16.659801 3319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:16.665403 3319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:16.666486 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:16.667483 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:16.676357 3319 server_base.cc:1034] running on GCE node
W20260929 14:42:16.676318 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:16.676295 3325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:16.676295 3327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:16.677301 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:16.678452 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:16.679625 3319 hybrid_clock.cc:648] HybridClock initialized: now 1790692936679611 us; error 40 us; skew 500 ppm
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:16.688127 3319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:16.689148 3319 webserver.cc:460] Webserver started at http://127.0.98.62:44579/ using document root <none> and password file <none>
I20260929 14:42:16.689733 3319 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:16.689796 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:16.689965 3319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:16.692081 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2"
format_stamp: "Formatted at 2026-09-29 14:42:16 on dist-test-slave-g8hp"
server_key: "50ba9d71331a2d0a3b5b2d8b7fd242a7"
server_key_iv: "ba2280c8ae9ee46a1ecfa8369049c0a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:16.692627 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2"
format_stamp: "Formatted at 2026-09-29 14:42:16 on dist-test-slave-g8hp"
server_key: "50ba9d71331a2d0a3b5b2d8b7fd242a7"
server_key_iv: "ba2280c8ae9ee46a1ecfa8369049c0a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:16.696483 3319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:16.699074 3334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:16.700430 3319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:16.700546 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2"
format_stamp: "Formatted at 2026-09-29 14:42:16 on dist-test-slave-g8hp"
server_key: "50ba9d71331a2d0a3b5b2d8b7fd242a7"
server_key_iv: "ba2280c8ae9ee46a1ecfa8369049c0a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:16.700678 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:16.718066 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:16.721490 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:16.721676 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:16.728412 3319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40297
I20260929 14:42:16.728713 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40297 every 8 connection(s)
I20260929 14:42:16.729421 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:16.732299 3387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:16.738026 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2: Bootstrap starting.
I20260929 14:42:16.739298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3319
I20260929 14:42:16.739404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:16.739627 392 external_mini_cluster.cc:1428] Setting key 7a90b75b19300720117107a155f8688d
I20260929 14:42:16.740692 3387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:16.741489 3387 log.cc:824] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:16.743722 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2: No bootstrap required, opened a new log
I20260929 14:42:16.747984 3387 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260929 14:42:16.748271 3387 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:16.748332 3387 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15ad22e24c64b7596d9f6b1a0d67ef2, State: Initialized, Role: FOLLOWER
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:16.748785 3387 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260929 14:42:16.748958 3387 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:16.749024 3387 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:16.749126 3387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:16.750039 3387 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260929 14:42:16.750368 3387 leader_election.cc:302] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15ad22e24c64b7596d9f6b1a0d67ef2; no voters:
I20260929 14:42:16.750694 3387 leader_election.cc:288] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:16.750785 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:16.750988 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [term 1 LEADER]: Becoming Leader. State: Replica: c15ad22e24c64b7596d9f6b1a0d67ef2, State: Running, Role: LEADER
I20260929 14:42:16.751366 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260929 14:42:16.752059 3387 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:16.753005 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260929 14:42:16.753141 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:16.753485 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15ad22e24c64b7596d9f6b1a0d67ef2. Latest consensus state: current_term: 1 leader_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15ad22e24c64b7596d9f6b1a0d67ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260929 14:42:16.753604 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:16.753923 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:16.755053 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:16.740638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.0.98.62:40297)
0929 14:42:16.741160 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:16.741177 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:16.741213 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:16.741878 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:16.741897 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:16.742244 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:16.742531 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:16.743423 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:16.744492 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:16.745752 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:16.746262 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:16.746397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:16.749449 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:16.749455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:16.749468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:16.749514 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:16.751361 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:16.751820 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:16.751827 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:16.751833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:16.751953 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:16.752230 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:16.752233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:16.752235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:16.752740 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:16.753563 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:16.753724 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:42:16.754029 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":148,"threads_started":1}
I20260929 14:42:16.757135 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:16.757774 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:42:16.763187 3401 catalog_manager.cc:1347] Generated new cluster ID: d088836eb76a4afc9047d02cd2f663ea
I20260929 14:42:16.763329 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:16.774181 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:16.769590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.0.98.62:40297)
0929 14:42:16.769709 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:16.769713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:16.769918 (+ 205us) server_negotiation.cc:407] Connection header received
0929 14:42:16.770048 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:16.770051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:16.770097 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:16.770158 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:16.770673 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:16.771224 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:16.772083 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:16.772335 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:16.772422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:16.773258 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:16.773262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:16.773265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:16.773742 (+ 477us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:16.773773 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:16.773790 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:16.773868 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:16.773969 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 14:42:16.774030 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:16.786492 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:16.787974 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:16.793848 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15ad22e24c64b7596d9f6b1a0d67ef2: Generated new TSK 0
I20260929 14:42:16.794720 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:16.805895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3319
2026-09-29T14:42:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1353 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3420](info): set up 2 sockets
Sep 29 14:42:16 dist-test-slave-g8hp krb5kdc[3420](info): commencing operation
krb5kdc: starting...
W20260929 14:42:17.760751 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:17 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:17.884792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:17.996006 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:17.996308 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:17.996366 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:18.000450 3433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:18.000535 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:18.000562 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:18.000583 3433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:18.000603 3433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:18.006067 3433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:18.007197 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:18.008249 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:18.016701 3433 server_base.cc:1034] running on GCE node
W20260929 14:42:18.016661 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.016777 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.016664 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:18.017580 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:18.018687 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:18.019848 3433 hybrid_clock.cc:648] HybridClock initialized: now 1790692938019834 us; error 33 us; skew 500 ppm
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:18.022787 3433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:18.023729 3433 webserver.cc:460] Webserver started at http://127.0.98.62:35409/ using document root <none> and password file <none>
I20260929 14:42:18.024293 3433 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:18.024353 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:18.024542 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:18.026674 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "c118248a91af499e87044321e3269cb9"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3b56a73dca04badf00e86787b46bc45c"
server_key_iv: "a2f032f40515bb8fa2df921a05bf0366"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.027232 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "c118248a91af499e87044321e3269cb9"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3b56a73dca04badf00e86787b46bc45c"
server_key_iv: "a2f032f40515bb8fa2df921a05bf0366"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.031138 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:18.033654 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.034986 3433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:18.035115 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "c118248a91af499e87044321e3269cb9"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3b56a73dca04badf00e86787b46bc45c"
server_key_iv: "a2f032f40515bb8fa2df921a05bf0366"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.035221 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:18.047161 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:18.050467 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:18.050665 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:18.058115 3433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37293
I20260929 14:42:18.058473 3500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37293 every 8 connection(s)
I20260929 14:42:18.059198 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:18.061434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3433
I20260929 14:42:18.061563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:18.061856 392 external_mini_cluster.cc:1428] Setting key 117c8d17e02e90f52ac24dad9e41ee76
I20260929 14:42:18.062716 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:18.069584 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9: Bootstrap starting.
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:18.072290 3501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:18.073077 3501 log.cc:824] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:18.075011 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9: No bootstrap required, opened a new log
I20260929 14:42:18.076817 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.063283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.98.62:37293)
0929 14:42:18.063697 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:18.063712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:18.063748 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:18.064400 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:18.064417 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:18.064782 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:18.065141 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:18.066102 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.067278 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.068549 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.069117 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.069244 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.072162 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:18.072176 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:18.072198 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:18.072233 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:18.073943 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.074398 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.074405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.074409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.074526 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.074782 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.074786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.074787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.075226 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:18.075403 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:18.075752 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 14:42:18.076055 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260929 14:42:18.078971 3501 raft_consensus.cc:348] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260929 14:42:18.079267 3501 raft_consensus.cc:374] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:18.079355 3501 raft_consensus.cc:729] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c118248a91af499e87044321e3269cb9, State: Initialized, Role: FOLLOWER
I20260929 14:42:18.079792 3501 consensus_queue.cc:260] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260929 14:42:18.079917 3501 raft_consensus.cc:388] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:18.079979 3501 raft_consensus.cc:482] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:18.080086 3501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:18.081055 3501 raft_consensus.cc:504] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260929 14:42:18.081380 3501 leader_election.cc:302] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c118248a91af499e87044321e3269cb9; no voters:
I20260929 14:42:18.081715 3501 leader_election.cc:288] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:18.081877 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:18.082077 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [term 1 LEADER]: Becoming Leader. State: Replica: c118248a91af499e87044321e3269cb9, State: Running, Role: LEADER
I20260929 14:42:18.082391 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } }
I20260929 14:42:18.082914 3501 sys_catalog.cc:564] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:18.083930 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c118248a91af499e87044321e3269cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } } }
I20260929 14:42:18.084059 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:18.084345 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c118248a91af499e87044321e3269cb9. Latest consensus state: current_term: 1 leader_uuid: "c118248a91af499e87044321e3269cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c118248a91af499e87044321e3269cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37293 } } }
I20260929 14:42:18.084430 3515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:18.084512 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:18.087718 3515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:18.093930 3515 catalog_manager.cc:1347] Generated new cluster ID: 7501d722f34f4d95a4afbda5252b5e25
I20260929 14:42:18.094007 3515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:18.130419 3515 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:18.131557 3515 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:18.139145 3515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c118248a91af499e87044321e3269cb9: Generated new TSK 0
I20260929 14:42:18.139909 3515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:18.171306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:18.309831 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:18.310125 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:18.310169 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:18.314276 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:18.314352 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:18.314414 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:18.319801 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:18.320842 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:18.321902 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:18.332038 3527 server_base.cc:1034] running on GCE node
W20260929 14:42:18.332113 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.332121 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.332310 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:18.332832 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:18.333540 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:18.334699 3527 hybrid_clock.cc:648] HybridClock initialized: now 1790692938334666 us; error 45 us; skew 500 ppm
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:18.337695 3527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:18.338703 3527 webserver.cc:460] Webserver started at http://127.0.98.1:37249/ using document root <none> and password file <none>
I20260929 14:42:18.339265 3527 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:18.339330 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:18.339635 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:18.341965 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "5696fb48ee1d43ca8231e8d30fac3245"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "20004c3ad210f46e934e6fc38dcf16c0"
server_key_iv: "e4180cfbf677dc03ae3eb62b15eeccb7"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.342563 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5696fb48ee1d43ca8231e8d30fac3245"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "20004c3ad210f46e934e6fc38dcf16c0"
server_key_iv: "e4180cfbf677dc03ae3eb62b15eeccb7"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.346449 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:18.349128 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.350543 3527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:18.350656 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "5696fb48ee1d43ca8231e8d30fac3245"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "20004c3ad210f46e934e6fc38dcf16c0"
server_key_iv: "e4180cfbf677dc03ae3eb62b15eeccb7"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.350771 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:18.364913 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:18.368405 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:18.368584 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:18.369249 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:18.370114 3527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:18.370160 3527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.370203 3527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:18.370232 3527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.378901 3527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39029
I20260929 14:42:18.378929 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39029 every 8 connection(s)
I20260929 14:42:18.379899 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:18.388684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3527
I20260929 14:42:18.388784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:18.389124 392 external_mini_cluster.cc:1428] Setting key 0a2a6610f83ade44b96445e9a7e53cea
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:18.395197 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.381867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47855 (local address 127.0.98.62:37293)
0929 14:42:18.382011 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:18.382016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:18.382937 (+ 921us) server_negotiation.cc:407] Connection header received
0929 14:42:18.384520 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:18.384524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:18.384583 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:18.384659 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:18.386006 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.386549 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.387731 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.387971 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.388068 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.391061 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:18.391066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:18.391069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:18.391098 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:18.392660 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.393397 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.393400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.393401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.393488 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.394010 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.394013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.394014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.394215 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:18.394307 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:18.394868 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 14:42:18.395020 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:18.396059 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.382289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.1:47855)
0929 14:42:18.382790 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:18.382835 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:18.384295 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:18.384845 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:18.384853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:18.385309 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:18.385838 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.385853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.386678 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.386681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.387606 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.387615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.388131 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.388135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.388304 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.389001 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:18.389023 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:18.390824 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:18.392796 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.392803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.392825 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.393258 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.393667 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.393669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.393670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.393897 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.394319 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:18.394326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:18.394613 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 14:42:18.394790 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 14:42:18.395164 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260929 14:42:18.397631 3656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260929 14:42:18.397897 3656 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:18.398445 3656 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:18.400074 3465 ts_manager.cc:194] Registered new tserver with Master: 5696fb48ee1d43ca8231e8d30fac3245 (127.0.98.1:39029)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:18.401585 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:18.415925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:18.527534 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:18.527849 3661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:18.527902 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:18.532197 3661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:18.532276 3661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:18.532341 3661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:18.538259 3661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:18.539453 3661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:18.540537 3661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:18.550598 3666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.550657 3669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.550657 3667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:18.550839 3661 server_base.cc:1034] running on GCE node
I20260929 14:42:18.551329 3661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:18.552091 3661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:18.553242 3661 hybrid_clock.cc:648] HybridClock initialized: now 1790692938553228 us; error 41 us; skew 500 ppm
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:18.556191 3661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:18.557163 3661 webserver.cc:460] Webserver started at http://127.0.98.2:39855/ using document root <none> and password file <none>
I20260929 14:42:18.557741 3661 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:18.557814 3661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:18.558001 3661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:18.560176 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "39ef54bc65cb4933b118c95e1eb11637"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "6dbce12ec94cf594ac0017a8cf3a39d7"
server_key_iv: "b0ded3c1ccf8a48d9cad9eb78beada9c"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.560729 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39ef54bc65cb4933b118c95e1eb11637"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "6dbce12ec94cf594ac0017a8cf3a39d7"
server_key_iv: "b0ded3c1ccf8a48d9cad9eb78beada9c"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.564545 3661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 14:42:18.567266 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.568599 3661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:18.568732 3661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "39ef54bc65cb4933b118c95e1eb11637"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "6dbce12ec94cf594ac0017a8cf3a39d7"
server_key_iv: "b0ded3c1ccf8a48d9cad9eb78beada9c"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.568845 3661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:18.585343 3661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:18.588796 3661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:18.589035 3661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:18.589720 3661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:18.590541 3661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:18.590587 3661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.590627 3661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:18.590641 3661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.599136 3661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37571
I20260929 14:42:18.599174 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37571 every 8 connection(s)
I20260929 14:42:18.600090 3661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:18.602401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3661
I20260929 14:42:18.602516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:18.602823 392 external_mini_cluster.cc:1428] Setting key 4796cb04e366dfbe862a3d82e51013fd
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:42:18.614917 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.602057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49393 (local address 127.0.98.62:37293)
0929 14:42:18.602216 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:18.602220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:18.603220 (+ 1000us) server_negotiation.cc:407] Connection header received
0929 14:42:18.604935 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:18.604938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:18.604995 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:18.605095 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:18.606469 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.606991 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.608163 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.608383 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.608476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.611292 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:18.611295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:18.611297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:18.611322 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:18.612702 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.613254 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.613258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.613260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.613350 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.613715 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.613718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.613720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.613944 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:18.614046 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:18.614614 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 14:42:18.614746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:18.615761 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.602537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.2:49393)
0929 14:42:18.603075 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:18.603121 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:18.604683 (+ 1562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:18.605278 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:18.605286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:18.605781 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:18.606313 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.606328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.607119 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.607122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.608037 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.608047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.608541 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.608545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.608711 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.609371 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:18.609389 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:18.611115 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:18.612814 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.612820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.612838 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.613158 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.613442 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.613445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.613447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.613625 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.614054 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:18.614062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:18.614363 (+ 301us) client_negotiation.cc:769] Sending connection context
0929 14:42:18.614553 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 14:42:18.614901 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260929 14:42:18.617195 3790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260929 14:42:18.617450 3790 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:18.617995 3790 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
I20260929 14:42:18.619179 3465 ts_manager.cc:194] Registered new tserver with Master: 39ef54bc65cb4933b118c95e1eb11637 (127.0.98.2:37571)
I20260929 14:42:18.619979 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:18.632700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:18.745067 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:18.745373 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:18.745417 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:18.749843 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:18.749913 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:18.749975 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:18.755522 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:18.756625 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:18.757707 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:18.767863 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.767866 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:18.767872 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:18.767923 3795 server_base.cc:1034] running on GCE node
I20260929 14:42:18.768746 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:18.769474 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:18.770660 3795 hybrid_clock.cc:648] HybridClock initialized: now 1790692938770619 us; error 55 us; skew 500 ppm
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:18.773751 3795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:18.774745 3795 webserver.cc:460] Webserver started at http://127.0.98.3:33649/ using document root <none> and password file <none>
I20260929 14:42:18.775273 3795 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:18.775332 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:18.775542 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:18.777619 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4eabb4882ad427dad1ffcd72222c324"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3419cc1b0db97c3c7ceb09bd5b32bf86"
server_key_iv: "af171c92f95030cec932026ed743b8f6"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.778141 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4eabb4882ad427dad1ffcd72222c324"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3419cc1b0db97c3c7ceb09bd5b32bf86"
server_key_iv: "af171c92f95030cec932026ed743b8f6"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.781931 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:18.785081 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.786770 3795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:18.786908 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "b4eabb4882ad427dad1ffcd72222c324"
format_stamp: "Formatted at 2026-09-29 14:42:18 on dist-test-slave-g8hp"
server_key: "3419cc1b0db97c3c7ceb09bd5b32bf86"
server_key_iv: "af171c92f95030cec932026ed743b8f6"
server_key_version: "encryptionkey@0"
I20260929 14:42:18.787034 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:18.804720 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:18.807996 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:18.808187 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:18.808799 3795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:18.809700 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:18.809747 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.809803 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:18.809831 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:18.817950 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39071
I20260929 14:42:18.818007 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39071 every 8 connection(s)
I20260929 14:42:18.818869 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:18.819154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3795
I20260929 14:42:18.819257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:18.819654 392 external_mini_cluster.cc:1428] Setting key 1e33e6312793561656c12397711895ac
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:18.833935 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.821059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36721 (local address 127.0.98.62:37293)
0929 14:42:18.821195 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:18.821200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:18.822044 (+ 844us) server_negotiation.cc:407] Connection header received
0929 14:42:18.823431 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:18.823434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:18.823481 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:18.823560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:18.824768 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.825416 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.826614 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.826848 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.826941 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.829658 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:18.829662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:18.829664 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:18.829691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:18.831205 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.831825 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.831829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.831830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.831923 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.832408 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.832412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.832414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.832630 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:18.832733 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:18.833617 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 14:42:18.833773 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:42:18.834830 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.821310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37293 (local address 127.0.98.3:36721)
0929 14:42:18.821895 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:18.821948 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:18.823216 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:18.823704 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:18.823711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:18.824098 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:18.824603 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.824618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.825528 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.825531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.826480 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:18.826490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.827000 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.827004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:18.827148 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.827739 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:18.827757 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:18.829464 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:18.831328 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.831334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.831348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.831704 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.832044 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:18.832048 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:18.832050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:18.832258 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:18.832766 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:18.832772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:18.833157 (+ 385us) client_negotiation.cc:769] Sending connection context
0929 14:42:18.833522 (+ 365us) client_negotiation.cc:240] Negotiation successful
0929 14:42:18.833898 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":137,"threads_started":1}
I20260929 14:42:18.836067 3924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37293
I20260929 14:42:18.836333 3924 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:18.836930 3924 heartbeater.cc:511] Master 127.0.98.62:37293 requested a full tablet report, sending...
I20260929 14:42:18.837955 3465 ts_manager.cc:194] Registered new tserver with Master: b4eabb4882ad427dad1ffcd72222c324 (127.0.98.3:39071)
I20260929 14:42:18.838598 3465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36721
I20260929 14:42:18.846210 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692938, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:18.871623 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:18.862920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.0.98.62:37293)
0929 14:42:18.863059 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:18.863062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:18.863158 (+ 96us) server_negotiation.cc:407] Connection header received
0929 14:42:18.863258 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:18.863260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:18.863305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:18.863382 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:18.864124 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.864691 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.865567 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:18.865776 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:18.865851 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:18.868404 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:18.868408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:18.868410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:18.868438 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:18.870040 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.870442 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.870448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.870449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.870521 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:18.870726 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:18.870728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:18.870729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:18.870940 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:18.871045 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:18.871292 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 14:42:18.871490 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 14:42:18.872051 3465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42646
I20260929 14:42:18.873080 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3527
I20260929 14:42:18.878120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3661
I20260929 14:42:18.883556 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3795
I20260929 14:42:18.888356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3433
2026-09-29T14:42:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2084 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3936](info): set up 2 sockets
Sep 29 14:42:18 dist-test-slave-g8hp krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260929 14:42:19.864828 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:19 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:19.989880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:20.102495 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:20.102831 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:20.102892 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:20.107299 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:20.107383 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:20.107419 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:20.107443 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:20.107472 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:20.113080 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:20.114156 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:20.115221 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:20.123836 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.123764 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.123764 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:20.123864 3949 server_base.cc:1034] running on GCE node
I20260929 14:42:20.124665 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:20.126120 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:20.127295 3949 hybrid_clock.cc:648] HybridClock initialized: now 1790692940127265 us; error 48 us; skew 500 ppm
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:20.130167 3949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:20.131269 3949 webserver.cc:460] Webserver started at http://127.0.98.62:39925/ using document root <none> and password file <none>
I20260929 14:42:20.131837 3949 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:20.131903 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:20.132071 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:20.134354 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "9e5027d5fff44319b87b98271885c09c"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "fa57b33890c34e95f00a2aaf660c7281"
server_key_iv: "5202b0e496fc30b2fbfad8a04967c850"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.134899 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e5027d5fff44319b87b98271885c09c"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "fa57b33890c34e95f00a2aaf660c7281"
server_key_iv: "5202b0e496fc30b2fbfad8a04967c850"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.138763 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 14:42:20.141227 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.142589 3949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:20.142704 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "9e5027d5fff44319b87b98271885c09c"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "fa57b33890c34e95f00a2aaf660c7281"
server_key_iv: "5202b0e496fc30b2fbfad8a04967c850"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.142820 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:20.162602 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:20.165583 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:20.165759 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:20.172382 3949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40395
I20260929 14:42:20.172417 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40395 every 8 connection(s)
I20260929 14:42:20.173362 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:20.176179 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:20.176589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 3949
I20260929 14:42:20.176692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:20.176982 392 external_mini_cluster.cc:1428] Setting key d07d9912bae964bfda2000854c2658ab
I20260929 14:42:20.183256 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c: Bootstrap starting.
I20260929 14:42:20.185690 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c: Neither blocks nor log segments found. Creating new log.
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:20.186414 4017 log.cc:824] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:20.188695 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c: No bootstrap required, opened a new log
I20260929 14:42:20.191991 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.178328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.0.98.62:40395)
0929 14:42:20.178751 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.178771 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.178814 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:42:20.179527 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.179550 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.179975 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.180363 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.181037 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.182528 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.183505 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.184179 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.184337 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.186778 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.186784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.186801 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.186832 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.188816 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.189288 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.189294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.189300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.189459 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.189712 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.189714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.189716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.190217 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.190421 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.190674 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.190997 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260929 14:42:20.193280 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 14:42:20.193509 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:20.193542 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5027d5fff44319b87b98271885c09c, State: Initialized, Role: FOLLOWER
I20260929 14:42:20.193966 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 14:42:20.194082 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:20.194140 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:20.194233 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:20.195180 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 14:42:20.195513 4017 leader_election.cc:302] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5027d5fff44319b87b98271885c09c; no voters:
I20260929 14:42:20.195804 4017 leader_election.cc:288] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:20.195909 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:20.196147 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [term 1 LEADER]: Becoming Leader. State: Replica: 9e5027d5fff44319b87b98271885c09c, State: Running, Role: LEADER
I20260929 14:42:20.196501 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } }
I20260929 14:42:20.197024 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:20.198216 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e5027d5fff44319b87b98271885c09c. Latest consensus state: current_term: 1 leader_uuid: "9e5027d5fff44319b87b98271885c09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } }
I20260929 14:42:20.198251 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e5027d5fff44319b87b98271885c09c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5027d5fff44319b87b98271885c09c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40395 } } }
I20260929 14:42:20.198354 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:20.198354 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:20.198782 4031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:20.202116 4031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:20.208091 4031 catalog_manager.cc:1347] Generated new cluster ID: cb0325641365404099368d9218fb759d
I20260929 14:42:20.208178 4031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:20.226364 4031 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:20.227355 4031 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:20.233650 4031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e5027d5fff44319b87b98271885c09c: Generated new TSK 0
I20260929 14:42:20.234375 4031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:20.268471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:20.382397 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:20.382786 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:20.382855 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:20.387027 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:20.387104 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:20.387250 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:20.393131 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:20.394253 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:20.395283 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:20.405085 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.405031 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.405031 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:20.405469 4043 server_base.cc:1034] running on GCE node
I20260929 14:42:20.405865 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:20.406486 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:20.407648 4043 hybrid_clock.cc:648] HybridClock initialized: now 1790692940407633 us; error 29 us; skew 500 ppm
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:20.410562 4043 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:20.411427 4043 webserver.cc:460] Webserver started at http://127.0.98.1:38103/ using document root <none> and password file <none>
I20260929 14:42:20.411947 4043 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:20.412003 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:20.412176 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:20.414198 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "52bad354da6e4b74bbb9ee7ffb505a55"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "d3acf9e8f6c3518031b4d50eb819852d"
server_key_iv: "4e30a1a6ac52e1d16a4e6891af051543"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.414711 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52bad354da6e4b74bbb9ee7ffb505a55"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "d3acf9e8f6c3518031b4d50eb819852d"
server_key_iv: "4e30a1a6ac52e1d16a4e6891af051543"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.418535 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:20.420936 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.422188 4043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:20.422297 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "52bad354da6e4b74bbb9ee7ffb505a55"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "d3acf9e8f6c3518031b4d50eb819852d"
server_key_iv: "4e30a1a6ac52e1d16a4e6891af051543"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.422385 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:20.443367 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:20.446583 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:20.446753 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:20.447347 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:20.448153 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:20.448199 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.448237 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:20.448251 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.456971 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34305
I20260929 14:42:20.457021 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34305 every 8 connection(s)
I20260929 14:42:20.457880 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:20.465523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4043
I20260929 14:42:20.465654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:20.465986 392 external_mini_cluster.cc:1428] Setting key f986d3c2dce97baa1b9eff249233af07
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:20.473460 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.460013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45409 (local address 127.0.98.62:40395)
0929 14:42:20.460140 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.460143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.461126 (+ 983us) server_negotiation.cc:407] Connection header received
0929 14:42:20.462507 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.462511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.462560 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.462638 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.464016 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.464571 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.466124 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.466385 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.466502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.469444 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.469449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.469452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.469482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.471007 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.471545 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.471549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.471550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.471649 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.472020 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.472023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.472026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.472287 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.472394 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.473116 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.473275 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:20.474229 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.460409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.1:45409)
0929 14:42:20.460971 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:20.461017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:20.462280 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:20.462900 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:20.462908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:20.463314 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:20.463832 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.463846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.464754 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.464757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.465985 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.465998 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.466593 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.466598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.466840 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.467465 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:20.467482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:20.469263 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:20.471128 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.471133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.471149 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.471429 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.471733 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.471735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.471736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.471919 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.472440 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:20.472446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:20.472795 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 14:42:20.473022 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:42:20.473324 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:20.475471 4172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 14:42:20.475741 4172 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:20.476257 4172 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:20.478060 3981 ts_manager.cc:194] Registered new tserver with Master: 52bad354da6e4b74bbb9ee7ffb505a55 (127.0.98.1:34305)
I20260929 14:42:20.479226 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:20.496161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:20.605882 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:20.606176 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:20.606240 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:20.610550 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:20.610634 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:20.610724 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:20.616319 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:20.617468 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:20.618513 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:20.628679 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.628683 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.628679 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:20.629060 4177 server_base.cc:1034] running on GCE node
I20260929 14:42:20.629592 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:20.630231 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:20.631412 4177 hybrid_clock.cc:648] HybridClock initialized: now 1790692940631383 us; error 40 us; skew 500 ppm
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:20.634286 4177 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:20.635233 4177 webserver.cc:460] Webserver started at http://127.0.98.2:43231/ using document root <none> and password file <none>
I20260929 14:42:20.635907 4177 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:20.635977 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:20.636168 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:20.638373 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e435ca6a8d34ec19769264d2bdde012"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "4938041a76d73b1b07d5bd462ce7f596"
server_key_iv: "02e43603b595b1655d580073f4ce6dff"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.638958 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e435ca6a8d34ec19769264d2bdde012"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "4938041a76d73b1b07d5bd462ce7f596"
server_key_iv: "02e43603b595b1655d580073f4ce6dff"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.642822 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:20.645447 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.646890 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:20.647011 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "6e435ca6a8d34ec19769264d2bdde012"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "4938041a76d73b1b07d5bd462ce7f596"
server_key_iv: "02e43603b595b1655d580073f4ce6dff"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.647126 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:20.662984 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:20.666618 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:20.666806 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:20.667420 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:20.668294 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:20.668342 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.668394 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:20.668423 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.677443 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35777
I20260929 14:42:20.677487 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35777 every 8 connection(s)
I20260929 14:42:20.678369 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:20.682566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4177
I20260929 14:42:20.682678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:20.682950 392 external_mini_cluster.cc:1428] Setting key 63122e305cfd11312dff976c06cddfbc
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:20.692289 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.680298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33075 (local address 127.0.98.62:40395)
0929 14:42:20.680476 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.680480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.681354 (+ 874us) server_negotiation.cc:407] Connection header received
0929 14:42:20.682731 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.682734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.682800 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.682878 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.684115 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.684646 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.685828 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.686048 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.686135 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.688740 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.688743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.688746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.688775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.690110 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.690592 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.690596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.690598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.690683 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.691144 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.691147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.691149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.691417 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.691525 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.692036 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.692162 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 14:42:20.693238 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.680709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.2:33075)
0929 14:42:20.681192 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:20.681237 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:20.682499 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:20.683078 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:20.683088 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:20.683461 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:20.683968 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.683982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.684774 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.684777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.685700 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.685710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.686187 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.686190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.686360 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.686935 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:20.686951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:20.688569 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:20.690221 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.690226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.690243 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.690497 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.690781 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.690784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.690786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.691021 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.691523 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:20.691529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:20.691802 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 14:42:20.691965 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 14:42:20.692300 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":150,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:42:20.694449 4306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 14:42:20.694717 4306 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:20.695284 4306 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
I20260929 14:42:20.696408 3981 ts_manager.cc:194] Registered new tserver with Master: 6e435ca6a8d34ec19769264d2bdde012 (127.0.98.2:35777)
I20260929 14:42:20.697278 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:20.712574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:20.824769 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:20.825102 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:20.825184 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:20.829731 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:20.829815 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:20.829900 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:20.835530 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:20.836674 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:20.837752 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:20.847702 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.847823 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:20.847707 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:20.847952 4311 server_base.cc:1034] running on GCE node
I20260929 14:42:20.848493 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:20.849195 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:20.850382 4311 hybrid_clock.cc:648] HybridClock initialized: now 1790692940850369 us; error 24 us; skew 500 ppm
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:20.853209 4311 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:20.854095 4311 webserver.cc:460] Webserver started at http://127.0.98.3:42827/ using document root <none> and password file <none>
I20260929 14:42:20.854673 4311 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:20.854743 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:20.854959 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:20.857067 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "bdf20c30ae4343f6b9010915d1bf2182"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "9ffb6c4aa464619bd2df80ae198938ce"
server_key_iv: "526f0874355317c9e159d00aade8cfbc"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.857631 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bdf20c30ae4343f6b9010915d1bf2182"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "9ffb6c4aa464619bd2df80ae198938ce"
server_key_iv: "526f0874355317c9e159d00aade8cfbc"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.861509 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:42:20.864094 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.865397 4311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260929 14:42:20.865512 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "bdf20c30ae4343f6b9010915d1bf2182"
format_stamp: "Formatted at 2026-09-29 14:42:20 on dist-test-slave-g8hp"
server_key: "9ffb6c4aa464619bd2df80ae198938ce"
server_key_iv: "526f0874355317c9e159d00aade8cfbc"
server_key_version: "encryptionkey@0"
I20260929 14:42:20.865624 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:20.908954 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:20.912210 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:20.912396 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:20.913058 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:20.913929 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:20.913977 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.914029 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:20.914058 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:20.922855 4311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36719
I20260929 14:42:20.922883 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36719 every 8 connection(s)
I20260929 14:42:20.923769 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:20.929232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4311
I20260929 14:42:20.929373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:20.929677 392 external_mini_cluster.cc:1428] Setting key b5d146608e4e4bb1f8f5aa8433a312e4
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:20.937968 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.925585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56213 (local address 127.0.98.62:40395)
0929 14:42:20.925744 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.925749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.926636 (+ 887us) server_negotiation.cc:407] Connection header received
0929 14:42:20.928073 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.928076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.928123 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.928194 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.929645 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.930282 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.931672 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.931937 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.932019 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.934493 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.934497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.934499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.934524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.935927 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.936495 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.936498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.936499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.936578 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.936944 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.936947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.936948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.937133 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.937208 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.937716 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.937843 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:42:20.938776 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.926023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40395 (local address 127.0.98.3:56213)
0929 14:42:20.926491 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:20.926538 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:20.927878 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:20.928426 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:20.928432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:20.928794 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:20.929500 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.929522 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.930407 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.930409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.931559 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:20.931570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.932062 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.932064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:20.932216 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.932795 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:20.932811 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:20.934306 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:20.936079 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.936085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.936100 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.936407 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.936654 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:20.936656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:20.936657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:20.936826 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:20.937215 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:20.937221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:20.937483 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 14:42:20.937640 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 14:42:20.937940 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260929 14:42:20.939982 4440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40395
I20260929 14:42:20.940205 4440 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:20.940737 4440 heartbeater.cc:511] Master 127.0.98.62:40395 requested a full tablet report, sending...
I20260929 14:42:20.941807 3981 ts_manager.cc:194] Registered new tserver with Master: bdf20c30ae4343f6b9010915d1bf2182 (127.0.98.3:36719)
I20260929 14:42:20.942382 3981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56213
I20260929 14:42:20.943331 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:20.969956 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.960092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.0.98.1:34305)
0929 14:42:20.960386 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.960401 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.960427 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:42:20.960516 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.960521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.960712 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.960896 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.961611 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.962566 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.963442 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.963851 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.963944 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.966377 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.966387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.966402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.966440 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.968021 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.968459 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.968464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.968466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.968547 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.968754 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.968756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.968758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.969058 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.969409 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.969627 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.969779 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
Sep 29 14:42:20 dist-test-slave-g8hp krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:20.981570 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:20.973063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.0.98.62:40395)
0929 14:42:20.973184 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:20.973188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:20.973333 (+ 145us) server_negotiation.cc:407] Connection header received
0929 14:42:20.973447 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:20.973450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:20.973494 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:20.973559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:20.974193 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.974729 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.975520 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:20.975774 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:20.975961 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:20.978593 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:20.978596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:20.978635 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:20.978664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:20.979954 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.980346 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.980349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.980350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.980422 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:20.980660 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:20.980663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:20.980665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:20.980946 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:20.981062 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:20.981294 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:42:20.981409 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 14:42:20.982007 3981 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41008
I20260929 14:42:20.982623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4043
I20260929 14:42:20.987634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4177
I20260929 14:42:20.992184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4311
I20260929 14:42:20.996752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 3949
2026-09-29T14:42:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2107 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:21 dist-test-slave-g8hp krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:21 dist-test-slave-g8hp krb5kdc[4452](info): set up 2 sockets
Sep 29 14:42:21 dist-test-slave-g8hp krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260929 14:42:21.956971 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:22.081508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:22.193939 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:22.194259 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:22.194304 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:22.198544 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:22.198618 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:22.198637 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:22.198653 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:22.198668 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:22.204260 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:22.205370 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:22.206377 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:22.214881 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.214886 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.214881 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.214990 4465 server_base.cc:1034] running on GCE node
I20260929 14:42:22.215847 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:22.217046 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:22.218227 4465 hybrid_clock.cc:648] HybridClock initialized: now 1790692942218202 us; error 49 us; skew 500 ppm
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:22.227850 4465 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:22.228694 4465 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:22.228777 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:22.229055 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:22.231330 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "f04521baf0ad4ef6b03824b9dade1343"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "a29bd305d5d43a8e38fb4110edc68414"
server_key_iv: "b8c3b780cd4f2f75a2ff85078273d95d"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.231931 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f04521baf0ad4ef6b03824b9dade1343"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "a29bd305d5d43a8e38fb4110edc68414"
server_key_iv: "b8c3b780cd4f2f75a2ff85078273d95d"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.236001 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:22.238754 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.240156 4465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:22.240266 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "f04521baf0ad4ef6b03824b9dade1343"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "a29bd305d5d43a8e38fb4110edc68414"
server_key_iv: "b8c3b780cd4f2f75a2ff85078273d95d"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.240365 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:22.250176 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:22.253809 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:22.253978 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:22.260931 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36485
I20260929 14:42:22.261301 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36485 every 8 connection(s)
I20260929 14:42:22.261721 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:22.264802 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:22.268198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4465
I20260929 14:42:22.268325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:22.268631 392 external_mini_cluster.cc:1428] Setting key 88b1f92ffffe10a412d16b3ac7ecae3e
I20260929 14:42:22.271009 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: Bootstrap starting.
I20260929 14:42:22.273733 4532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:22.274536 4532 log.cc:824] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:22.276932 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: No bootstrap required, opened a new log
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:22.280920 4532 raft_consensus.cc:348] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260929 14:42:22.281242 4532 raft_consensus.cc:374] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:22.281314 4532 raft_consensus.cc:729] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04521baf0ad4ef6b03824b9dade1343, State: Initialized, Role: FOLLOWER
I20260929 14:42:22.281802 4532 consensus_queue.cc:260] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260929 14:42:22.281941 4532 raft_consensus.cc:388] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:22.282009 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:22.282119 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:22.283061 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260929 14:42:22.283411 4532 leader_election.cc:302] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f04521baf0ad4ef6b03824b9dade1343; no voters:
I20260929 14:42:22.283747 4532 leader_election.cc:288] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:22.283809 4537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:22.284041 4537 raft_consensus.cc:686] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [term 1 LEADER]: Becoming Leader. State: Replica: f04521baf0ad4ef6b03824b9dade1343, State: Running, Role: LEADER
I20260929 14:42:22.284058 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:22.270237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.98.62:36485)
0929 14:42:22.270673 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:22.270689 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:22.270726 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:42:22.271441 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:22.271460 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:22.271838 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:22.272149 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:22.273104 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.274361 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.275478 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.276044 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.276185 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:22.278992 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:22.278998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:22.279013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:22.279141 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:22.280904 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.281379 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.281386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.281392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.281524 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.281828 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.281832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.281834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.282363 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:22.282571 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:22.282852 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 14:42:22.283154 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260929 14:42:22.284426 4537 consensus_queue.cc:237] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260929 14:42:22.285166 4532 sys_catalog.cc:564] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:22.286339 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f04521baf0ad4ef6b03824b9dade1343. Latest consensus state: current_term: 1 leader_uuid: "f04521baf0ad4ef6b03824b9dade1343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } } }
I20260929 14:42:22.286468 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:22.287498 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f04521baf0ad4ef6b03824b9dade1343" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f04521baf0ad4ef6b03824b9dade1343" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } } }
I20260929 14:42:22.287642 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343 [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:22.290200 4552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:22.290288 4552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:22.290343 4546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:22.291056 4546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:22.296983 4546 catalog_manager.cc:1347] Generated new cluster ID: f14724b58b164c179f2e442a2864072f
I20260929 14:42:22.297160 4546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:22.308116 4546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:22.309465 4546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:22.315013 4546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: Generated new TSK 0
I20260929 14:42:22.315920 4546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:22.348191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:22.460536 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:22.460821 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:22.460883 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:22.465164 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:22.465232 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:22.465298 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:22.470772 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:22.471851 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:22.472909 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:22.482460 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.482543 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.482678 4558 server_base.cc:1034] running on GCE node
W20260929 14:42:22.482604 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.483178 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:22.483879 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:22.485031 4558 hybrid_clock.cc:648] HybridClock initialized: now 1790692942485016 us; error 37 us; skew 500 ppm
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:22.487980 4558 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:22.488775 4558 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:22.488847 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:22.489105 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:22.491163 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fe9935564694511b33e2bc056d11d75"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "0386b1751fc4a31ce63beef89eae7d9a"
server_key_iv: "61ee3778e2a8f9387136243f5fa75777"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.491727 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fe9935564694511b33e2bc056d11d75"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "0386b1751fc4a31ce63beef89eae7d9a"
server_key_iv: "61ee3778e2a8f9387136243f5fa75777"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.495509 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 14:42:22.498208 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.499447 4558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:22.499594 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "2fe9935564694511b33e2bc056d11d75"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "0386b1751fc4a31ce63beef89eae7d9a"
server_key_iv: "61ee3778e2a8f9387136243f5fa75777"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.499696 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:22.532953 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:22.536305 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:22.536499 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:22.537122 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:22.537997 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:22.538046 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.538100 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:22.538129 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.546823 4558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41413
I20260929 14:42:22.546864 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41413 every 8 connection(s)
I20260929 14:42:22.547422 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:22.554565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4558
I20260929 14:42:22.554687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:22.555047 392 external_mini_cluster.cc:1428] Setting key 29ac9b5f35ee8936cc11c4d2b48457b0
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:22.562114 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:22.549235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43887 (local address 127.0.98.62:36485)
0929 14:42:22.549375 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:22.549380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:22.550299 (+ 919us) server_negotiation.cc:407] Connection header received
0929 14:42:22.551613 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:22.551616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:22.551663 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:22.551744 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:22.553081 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.553641 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.554886 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.555230 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.555337 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:22.558159 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:22.558162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:22.558164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:22.558190 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:22.559602 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.560221 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.560224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.560226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.560329 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.560737 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.560740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.560741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.560993 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:22.561142 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:22.561759 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 14:42:22.561876 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:42:22.563329 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:22.549685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36485 (local address 127.0.98.1:43887)
0929 14:42:22.550146 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:22.550192 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:22.551399 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:22.551903 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:22.551909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:22.552366 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:22.552883 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:22.552903 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.553767 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.553770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:22.554761 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:22.554771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.555380 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.555383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:22.555616 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:22.556190 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:22.556206 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:22.557880 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:22.559721 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:22.559727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:22.559744 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:22.560101 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:22.560423 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:22.560426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:22.560428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:22.560616 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:22.561103 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:22.561111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:22.561464 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 14:42:22.561705 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 14:42:22.562105 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:22.564785 4686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36485
I20260929 14:42:22.565071 4686 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:22.565546 4686 heartbeater.cc:511] Master 127.0.98.62:36485 requested a full tablet report, sending...
I20260929 14:42:22.567348 4496 ts_manager.cc:194] Registered new tserver with Master: 2fe9935564694511b33e2bc056d11d75 (127.0.98.1:41413)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:22.568792 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:22.584445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:22.694386 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:22.694669 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:22.694734 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:22.698845 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:22.698923 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:22.699007 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:22.704818 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:22.705936 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:22.707019 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:22.716768 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.716774 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.716776 4691 server_base.cc:1034] running on GCE node
W20260929 14:42:22.716795 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.717520 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:22.718195 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:22.719385 4691 hybrid_clock.cc:648] HybridClock initialized: now 1790692942719360 us; error 37 us; skew 500 ppm
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:22.722337 4691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:22.723060 4691 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:22.723138 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:22.723330 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:22.725414 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a6a085f2b6c459f94a77b8a7d282ecf"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "c3e35e63c53c208d3b18f6e5953c4e2b"
server_key_iv: "f18e9f6b0f154dd70e4140b711f906c3"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.725939 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a6a085f2b6c459f94a77b8a7d282ecf"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "c3e35e63c53c208d3b18f6e5953c4e2b"
server_key_iv: "f18e9f6b0f154dd70e4140b711f906c3"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.729794 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:42:22.732299 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.733741 4691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:22.733855 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "7a6a085f2b6c459f94a77b8a7d282ecf"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "c3e35e63c53c208d3b18f6e5953c4e2b"
server_key_iv: "f18e9f6b0f154dd70e4140b711f906c3"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.733958 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:22.757985 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:22.761317 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:22.761503 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:22.762086 4691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:22.762899 4691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:22.762948 4691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.763000 4691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:22.763028 4691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.771694 4691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44285
I20260929 14:42:22.771756 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44285 every 8 connection(s)
I20260929 14:42:22.772305 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:22.781410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4691
I20260929 14:42:22.781534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:22.781800 392 external_mini_cluster.cc:1428] Setting key e9c97449ef160aa71132dccfbf166401
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:22.787192 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:22.774235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54447 (local address 127.0.98.62:36485)
0929 14:42:22.774398 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:22.774402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:22.775320 (+ 918us) server_negotiation.cc:407] Connection header received
0929 14:42:22.776823 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:22.776826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:22.776908 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:22.776983 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:22.778233 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.778767 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.779892 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.780118 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.780208 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:22.782896 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:22.782899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:22.782902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:22.782929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:22.784726 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.785402 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.785406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.785408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.785551 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:22.785955 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:22.785958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:22.785960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:22.786179 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:22.786274 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:22.786892 (+ 618us) server_negotiation.cc:299] Negotiation successful
0929 14:42:22.787026 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 14:42:22.788285 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:22.774666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36485 (local address 127.0.98.2:54447)
0929 14:42:22.775161 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:22.775205 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:22.776611 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:22.777167 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:22.777175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:22.777572 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:22.778088 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:22.778102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.778885 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.778888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:22.779774 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:22.779783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:22.780262 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:22.780266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:22.780424 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:22.781120 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:22.781137 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:22.782738 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:22.784857 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:22.784902 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:22.784928 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:22.785294 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:22.785658 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:22.785660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:22.785661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:22.785862 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:22.786294 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:22.786302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:22.786651 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 14:42:22.786834 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 14:42:22.787190 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":113,"threads_started":1}
I20260929 14:42:22.789587 4819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36485
I20260929 14:42:22.789827 4819 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:22.790285 4819 heartbeater.cc:511] Master 127.0.98.62:36485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:22.791365 4496 ts_manager.cc:194] Registered new tserver with Master: 7a6a085f2b6c459f94a77b8a7d282ecf (127.0.98.2:44285)
I20260929 14:42:22.791939 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:22.812088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:22.924746 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:22.925122 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:22.925173 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:22.929430 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:22.929502 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:22.929566 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:22.935012 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:22.936090 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:22.937158 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:22.947001 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.947002 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:22.947002 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:22.947260 4824 server_base.cc:1034] running on GCE node
I20260929 14:42:22.947960 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:22.948617 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:22.949779 4824 hybrid_clock.cc:648] HybridClock initialized: now 1790692942949687 us; error 107 us; skew 500 ppm
Sep 29 14:42:22 dist-test-slave-g8hp krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:22.952698 4824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:22.953498 4824 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:22.953570 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:22.953761 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:22.955922 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "a28a0a80979d4811bf8461cb929addb3"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "ca05d79934b6020fc37d8f8818b65b4e"
server_key_iv: "ec3eb3ddefa2c9b947bc77ec2e7336ad"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.956501 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a28a0a80979d4811bf8461cb929addb3"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "ca05d79934b6020fc37d8f8818b65b4e"
server_key_iv: "ec3eb3ddefa2c9b947bc77ec2e7336ad"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.960407 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 14:42:22.963375 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:22.964792 4824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:22.964994 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "a28a0a80979d4811bf8461cb929addb3"
format_stamp: "Formatted at 2026-09-29 14:42:22 on dist-test-slave-g8hp"
server_key: "ca05d79934b6020fc37d8f8818b65b4e"
server_key_iv: "ec3eb3ddefa2c9b947bc77ec2e7336ad"
server_key_version: "encryptionkey@0"
I20260929 14:42:22.965139 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:23.014832 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:23.018042 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:23.018224 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:23.018817 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:23.019632 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:23.019726 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.019790 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:23.019817 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.028213 4824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45403
I20260929 14:42:23.028242 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45403 every 8 connection(s)
I20260929 14:42:23.028813 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:23.029120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4824
I20260929 14:42:23.029248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:23.029551 392 external_mini_cluster.cc:1428] Setting key e02ffdb31e9c2825e957a5a2329c7164
Sep 29 14:42:23 dist-test-slave-g8hp krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:23.043431 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.030851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39411 (local address 127.0.98.62:36485)
0929 14:42:23.031024 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.031029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.032013 (+ 984us) server_negotiation.cc:407] Connection header received
0929 14:42:23.033442 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.033445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.033493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.033566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:23.034820 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.035349 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.036478 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.036714 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.036802 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.039668 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:23.039671 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:23.039673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:23.039703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:23.041372 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:23.041915 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:23.041918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:23.041920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:23.042004 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:23.042328 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:23.042331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:23.042333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:23.042527 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:23.042612 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.043137 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.043270 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:42:23.044428 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.031388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36485 (local address 127.0.98.3:39411)
0929 14:42:23.031860 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.031911 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.033226 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.033716 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.033724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.034189 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:23.034673 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.034685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.035453 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.035456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.036335 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.036344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.036849 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.036851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.037106 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.037657 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:23.037677 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:23.039425 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:23.041499 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:23.041506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:23.041525 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:23.041805 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:23.042090 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:23.042093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:23.042094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:23.042239 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:23.042626 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:23.042634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:23.042897 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.043078 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.043420 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260929 14:42:23.045732 4952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36485
I20260929 14:42:23.046085 4952 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:23.046579 4952 heartbeater.cc:511] Master 127.0.98.62:36485 requested a full tablet report, sending...
I20260929 14:42:23.047703 4496 ts_manager.cc:194] Registered new tserver with Master: a28a0a80979d4811bf8461cb929addb3 (127.0.98.3:45403)
I20260929 14:42:23.048348 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39411
I20260929 14:42:23.056020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:42:23.065706 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.058115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.0.98.62:36485)
0929 14:42:23.058250 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.058253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.058347 (+ 94us) server_negotiation.cc:407] Connection header received
0929 14:42:23.058484 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.058487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.058535 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.058620 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:23.059282 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.059840 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.060788 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.061135 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.061260 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.062732 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:23.062735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:23.062737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:23.062766 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:23.063930 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:23.064390 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:23.064393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:23.064394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:23.064469 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:23.064718 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:23.064721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:23.064723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:23.065013 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:23.065203 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.065435 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.065547 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:42:23.068853 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:23.072204 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:23.088620 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.081996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45403 (local address 127.0.0.1:45234)
0929 14:42:23.082759 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.082819 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.082953 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.083904 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.083911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.083932 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:23.084081 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.084088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085286 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.085290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.086778 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.086788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.087930 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.087933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.088022 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.088057 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.088264 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.088414 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":41,"thread_start_us":101,"threads_started":1}
I20260929 14:42:23.088620 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.082801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41413 (local address 127.0.0.1:56938)
0929 14:42:23.083371 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.083385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.083476 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.084130 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.084133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.084152 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:23.084286 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.084293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085743 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.085747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.087118 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.087129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.088211 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.088214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.088297 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.088319 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.088381 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.088435 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":105,"thread_start_us":103,"threads_started":1}
I20260929 14:42:23.089499 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.082195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.0.98.3:45403)
0929 14:42:23.083439 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.083449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.083472 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:42:23.083547 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.083553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.083749 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.083902 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:23.084189 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085086 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.086898 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.087792 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.087913 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.088094 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.089223 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.089342 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":161,"threads_started":1}
I20260929 14:42:23.089525 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.083247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.0.98.1:41413)
0929 14:42:23.083624 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.083630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.083648 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:42:23.083709 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.083713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.083849 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.083989 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:23.084405 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085605 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.087270 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.088077 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.088182 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.088322 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.088418 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.088535 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":85,"threads_started":1}
I20260929 14:42:23.089766 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.082420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.0.1:58758)
0929 14:42:23.083356 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.083370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.083458 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.084103 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.084106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.084127 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:23.084241 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.084246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085825 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.085828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.087118 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.087129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.089370 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.089374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.089466 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.089488 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.089558 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.089636 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":85,"thread_start_us":87,"threads_started":1}
I20260929 14:42:23.090883 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.082880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.0.98.2:44285)
0929 14:42:23.083244 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.083257 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.083548 (+ 291us) server_negotiation.cc:407] Connection header received
0929 14:42:23.083632 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.083638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.083839 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.083989 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:23.084345 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.085698 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.088184 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.089070 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.089173 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.089310 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.090607 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.090728 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260929 14:42:23.092466 4620 tablet_service.cc:1467] Processing CreateTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf (DEFAULT_TABLE table=test-table [id=489f02a137714966b4a302973408b1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:23.092518 4886 tablet_service.cc:1467] Processing CreateTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf (DEFAULT_TABLE table=test-table [id=489f02a137714966b4a302973408b1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:23.093617 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e7883503614ba1bd2a44d327a3ecaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:23.093612 4620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e7883503614ba1bd2a44d327a3ecaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:23.093544 4753 tablet_service.cc:1467] Processing CreateTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf (DEFAULT_TABLE table=test-table [id=489f02a137714966b4a302973408b1ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:23.094486 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60e7883503614ba1bd2a44d327a3ecaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:23.099762 4969 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: Bootstrap starting.
I20260929 14:42:23.101166 4970 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Bootstrap starting.
I20260929 14:42:23.101356 4971 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Bootstrap starting.
I20260929 14:42:23.102145 4969 tablet_bootstrap.cc:654] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:23.102888 4969 log.cc:824] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:23.103389 4970 tablet_bootstrap.cc:654] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:23.103389 4971 tablet_bootstrap.cc:654] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:23.104645 4971 log.cc:824] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:23.104645 4970 log.cc:824] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:23.105439 4969 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: No bootstrap required, opened a new log
I20260929 14:42:23.105648 4969 ts_tablet_manager.cc:1397] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:23.106657 4971 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: No bootstrap required, opened a new log
I20260929 14:42:23.106849 4970 tablet_bootstrap.cc:492] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: No bootstrap required, opened a new log
I20260929 14:42:23.106843 4971 ts_tablet_manager.cc:1397] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:23.107002 4970 ts_tablet_manager.cc:1397] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:23.109201 4969 raft_consensus.cc:348] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.109443 4969 raft_consensus.cc:374] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:23.109525 4969 raft_consensus.cc:729] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe9935564694511b33e2bc056d11d75, State: Initialized, Role: FOLLOWER
I20260929 14:42:23.109995 4969 consensus_queue.cc:260] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.110719 4971 raft_consensus.cc:348] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.110671 4970 raft_consensus.cc:348] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.110877 4686 heartbeater.cc:503] Master 127.0.98.62:36485 was elected leader, sending a full tablet report...
I20260929 14:42:23.110945 4970 raft_consensus.cc:374] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:23.111017 4971 raft_consensus.cc:374] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:23.111042 4970 raft_consensus.cc:729] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28a0a80979d4811bf8461cb929addb3, State: Initialized, Role: FOLLOWER
I20260929 14:42:23.111104 4971 raft_consensus.cc:729] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6a085f2b6c459f94a77b8a7d282ecf, State: Initialized, Role: FOLLOWER
I20260929 14:42:23.111548 4971 consensus_queue.cc:260] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.111548 4970 consensus_queue.cc:260] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.112473 4969 ts_tablet_manager.cc:1428] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:42:23.112528 4970 ts_tablet_manager.cc:1428] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:42:23.112528 4971 ts_tablet_manager.cc:1428] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:42:23.112948 4819 heartbeater.cc:503] Master 127.0.98.62:36485 was elected leader, sending a full tablet report...
I20260929 14:42:23.112948 4952 heartbeater.cc:503] Master 127.0.98.62:36485 was elected leader, sending a full tablet report...
I20260929 14:42:23.172049 4975 raft_consensus.cc:482] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:23.172261 4975 raft_consensus.cc:504] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.173561 4975 leader_election.cc:288] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a28a0a80979d4811bf8461cb929addb3 (127.0.98.3:45403), 7a6a085f2b6c459f94a77b8a7d282ecf (127.0.98.2:44285)
I20260929 14:42:23.177546 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.173906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42401 (local address 127.0.98.2:44285)
0929 14:42:23.174031 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.174035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.174342 (+ 307us) server_negotiation.cc:407] Connection header received
0929 14:42:23.174458 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.174461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.174508 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.174569 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:23.174878 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.175366 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.176509 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.176966 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.177060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.177123 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.177343 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.177414 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:42:23.177553 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.173905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37687 (local address 127.0.98.3:45403)
0929 14:42:23.174026 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.174031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.174288 (+ 257us) server_negotiation.cc:407] Connection header received
0929 14:42:23.174417 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.174420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.174467 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.174585 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:23.174873 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.175385 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.176488 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.176959 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.177053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.177117 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.177342 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.177411 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:42:23.177584 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.173840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45403 (local address 127.0.98.1:37687)
0929 14:42:23.174185 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.174197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.174316 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.174597 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.174600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.174615 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:23.174773 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.174779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.175495 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.175499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.176372 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.176381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.177074 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.177078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.177223 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.177256 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.177320 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.177409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":41,"thread_start_us":125,"threads_started":1}
I20260929 14:42:23.177590 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.173840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44285 (local address 127.0.98.1:42401)
0929 14:42:23.174244 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.174255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.174358 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.174587 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.174591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.174605 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:23.174777 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.174783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.175515 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.175518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.176388 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:23.176397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.177074 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.177078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:23.177222 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.177257 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.177319 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.177406 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":108,"spinlock_wait_cycles":19200,"thread_start_us":87,"threads_started":1}
I20260929 14:42:23.178450 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e7883503614ba1bd2a44d327a3ecaf" candidate_uuid: "2fe9935564694511b33e2bc056d11d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28a0a80979d4811bf8461cb929addb3" is_pre_election: true
I20260929 14:42:23.178507 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e7883503614ba1bd2a44d327a3ecaf" candidate_uuid: "2fe9935564694511b33e2bc056d11d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" is_pre_election: true
I20260929 14:42:23.178785 4773 raft_consensus.cc:2454] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe9935564694511b33e2bc056d11d75 in term 0.
I20260929 14:42:23.178785 4906 raft_consensus.cc:2454] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fe9935564694511b33e2bc056d11d75 in term 0.
I20260929 14:42:23.179294 4574 leader_election.cc:302] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe9935564694511b33e2bc056d11d75, 7a6a085f2b6c459f94a77b8a7d282ecf; no voters:
I20260929 14:42:23.179554 4975 raft_consensus.cc:2781] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:23.179636 4975 raft_consensus.cc:482] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:23.179687 4975 raft_consensus.cc:3037] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:23.180667 4975 raft_consensus.cc:504] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.181057 4975 leader_election.cc:288] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [CANDIDATE]: Term 1 election: Requested vote from peers a28a0a80979d4811bf8461cb929addb3 (127.0.98.3:45403), 7a6a085f2b6c459f94a77b8a7d282ecf (127.0.98.2:44285)
I20260929 14:42:23.181447 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e7883503614ba1bd2a44d327a3ecaf" candidate_uuid: "2fe9935564694511b33e2bc056d11d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf"
I20260929 14:42:23.181493 4906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60e7883503614ba1bd2a44d327a3ecaf" candidate_uuid: "2fe9935564694511b33e2bc056d11d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28a0a80979d4811bf8461cb929addb3"
I20260929 14:42:23.181619 4906 raft_consensus.cc:3037] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:23.181619 4773 raft_consensus.cc:3037] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:23.182981 4906 raft_consensus.cc:2454] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe9935564694511b33e2bc056d11d75 in term 1.
I20260929 14:42:23.182981 4773 raft_consensus.cc:2454] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fe9935564694511b33e2bc056d11d75 in term 1.
I20260929 14:42:23.183314 4574 leader_election.cc:302] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe9935564694511b33e2bc056d11d75, 7a6a085f2b6c459f94a77b8a7d282ecf; no voters:
I20260929 14:42:23.183492 4975 raft_consensus.cc:2781] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:23.183679 4975 raft_consensus.cc:686] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe9935564694511b33e2bc056d11d75, State: Running, Role: LEADER
I20260929 14:42:23.184033 4975 consensus_queue.cc:237] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } }
I20260929 14:42:23.187322 4495 catalog_manager.cc:5495] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fe9935564694511b33e2bc056d11d75 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2fe9935564694511b33e2bc056d11d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fe9935564694511b33e2bc056d11d75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:23.211651 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.207871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.0.98.1:41413)
0929 14:42:23.208018 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.208021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.208041 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:42:23.208118 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.208120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.208175 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.208278 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:23.208543 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.209242 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.210110 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:23.210374 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:23.210463 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:23.210767 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:23.211196 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:23.211291 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.211408 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.211498 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:42:23.217732 4773 raft_consensus.cc:1261] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 1 FOLLOWER]: Refusing update from remote peer 2fe9935564694511b33e2bc056d11d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:23.217747 4906 raft_consensus.cc:1261] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 1 FOLLOWER]: Refusing update from remote peer 2fe9935564694511b33e2bc056d11d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:23.218415 4980 consensus_queue.cc:1035] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28a0a80979d4811bf8461cb929addb3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:23.218582 4975 consensus_queue.cc:1035] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a6a085f2b6c459f94a77b8a7d282ecf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:23.235046 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:23.235220 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:23.237546 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f04521baf0ad4ef6b03824b9dade1343: Sending DeleteTablet for 3 replicas of tablet 60e7883503614ba1bd2a44d327a3ecaf
I20260929 14:42:23.238431 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45234
I20260929 14:42:23.238467 4620 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56938
I20260929 14:42:23.238973 4753 tablet_service.cc:1514] Processing DeleteTablet for tablet 60e7883503614ba1bd2a44d327a3ecaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58758
I20260929 14:42:23.239344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4558
I20260929 14:42:23.239454 4992 tablet_replica.cc:330] T 60e7883503614ba1bd2a44d327a3ecaf P 2fe9935564694511b33e2bc056d11d75: stopping tablet replica
I20260929 14:42:23.239506 4994 tablet_replica.cc:330] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: stopping tablet replica
I20260929 14:42:23.239781 4994 raft_consensus.cc:2229] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:23.239950 4993 tablet_replica.cc:330] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: stopping tablet replica
I20260929 14:42:23.240096 4994 raft_consensus.cc:2258] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:23.240195 4993 raft_consensus.cc:2229] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:23.240473 4993 raft_consensus.cc:2258] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:23.241789 4994 ts_tablet_manager.cc:1905] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:23.241853 4993 ts_tablet_manager.cc:1905] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:23.244086 4993 ts_tablet_manager.cc:1918] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:23.244184 4993 log.cc:1196] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/60e7883503614ba1bd2a44d327a3ecaf
I20260929 14:42:23.244447 4993 ts_tablet_manager.cc:1939] T 60e7883503614ba1bd2a44d327a3ecaf P a28a0a80979d4811bf8461cb929addb3: Deleting consensus metadata
I20260929 14:42:23.244364 4994 ts_tablet_manager.cc:1918] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:23.244513 4994 log.cc:1196] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/60e7883503614ba1bd2a44d327a3ecaf
I20260929 14:42:23.244781 4994 ts_tablet_manager.cc:1939] T 60e7883503614ba1bd2a44d327a3ecaf P 7a6a085f2b6c459f94a77b8a7d282ecf: Deleting consensus metadata
I20260929 14:42:23.245639 4483 catalog_manager.cc:4841] TS a28a0a80979d4811bf8461cb929addb3 (127.0.98.3:45403): tablet 60e7883503614ba1bd2a44d327a3ecaf (table test-table [id=489f02a137714966b4a302973408b1ef]) successfully deleted
I20260929 14:42:23.245889 4481 catalog_manager.cc:4841] TS 7a6a085f2b6c459f94a77b8a7d282ecf (127.0.98.2:44285): tablet 60e7883503614ba1bd2a44d327a3ecaf (table test-table [id=489f02a137714966b4a302973408b1ef]) successfully deleted
W20260929 14:42:23.246845 4483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41413 (error 108)
I20260929 14:42:23.247035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4691
I20260929 14:42:23.248036 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.247780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41413 (local address 127.0.0.1:56958)
0929 14:42:23.247899 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.247962 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 14:42:23.248188 4483 catalog_manager.cc:4584] TS 2fe9935564694511b33e2bc056d11d75 (127.0.98.1:41413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60e7883503614ba1bd2a44d327a3ecaf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41413: connect: Connection refused (error 111)
I20260929 14:42:23.252521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4824
I20260929 14:42:23.257261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4465
2026-09-29T14:42:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2262 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T14:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:23Z Disabled control of system clock
I20260929 14:42:23.287393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41013
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:42:23.397534 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:23.397846 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:23.397884 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:23.402204 4999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:23.402282 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:23.402297 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:23.402313 4999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:23.402328 4999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:23.407855 4999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:23.409372 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:23.410907 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:23.421595 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:23.421620 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:23.422053 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:23.422186 4999 server_base.cc:1034] running on GCE node
I20260929 14:42:23.422852 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:23.424409 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:23.425609 4999 hybrid_clock.cc:648] HybridClock initialized: now 1790692943425578 us; error 42 us; skew 500 ppm
I20260929 14:42:23.427894 4999 webserver.cc:460] Webserver started at http://127.0.98.62:38089/ using document root <none> and password file <none>
I20260929 14:42:23.428623 4999 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:23.428717 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:23.429035 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:23.431965 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "fb578c898e0f4bd5bbc541a4ab1f207f"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "82f9d5530099c8cd10cd362e781abdd3"
server_key_iv: "09da91ab7f725c8a78b4cf262dbf0bb9"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.432675 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb578c898e0f4bd5bbc541a4ab1f207f"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "82f9d5530099c8cd10cd362e781abdd3"
server_key_iv: "09da91ab7f725c8a78b4cf262dbf0bb9"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.437623 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 14:42:23.440737 5013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.442368 4999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:23.442500 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "fb578c898e0f4bd5bbc541a4ab1f207f"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "82f9d5530099c8cd10cd362e781abdd3"
server_key_iv: "09da91ab7f725c8a78b4cf262dbf0bb9"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.442631 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:23.447513 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:23.448140 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:23.448307 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:23.456056 4999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41013
I20260929 14:42:23.456099 5065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41013 every 8 connection(s)
I20260929 14:42:23.457075 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:23.460219 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:23.463165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 4999
I20260929 14:42:23.463284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:23.463526 392 external_mini_cluster.cc:1428] Setting key a8d3ff792ab3e2e73ae71c04523097f9
I20260929 14:42:23.466123 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: Bootstrap starting.
I20260929 14:42:23.468660 5066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:23.469497 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.464926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.0.98.62:41013)
0929 14:42:23.465563 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.465578 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.465614 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:23.466361 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.466381 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.466761 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.467068 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:23.467995 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:23.468000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:23.468014 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:23.468032 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:23.468096 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:23.468113 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:23.468204 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.468352 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.468611 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260929 14:42:23.469491 5066 log.cc:824] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:23.472088 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: No bootstrap required, opened a new log
I20260929 14:42:23.475658 5066 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } }
I20260929 14:42:23.475883 5066 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:23.475926 5066 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb578c898e0f4bd5bbc541a4ab1f207f, State: Initialized, Role: FOLLOWER
I20260929 14:42:23.476312 5066 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } }
I20260929 14:42:23.476444 5066 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:23.476483 5066 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:23.476552 5066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:23.477468 5066 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } }
I20260929 14:42:23.477790 5066 leader_election.cc:302] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb578c898e0f4bd5bbc541a4ab1f207f; no voters:
I20260929 14:42:23.478075 5066 leader_election.cc:288] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:23.478201 5071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:23.478395 5071 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [term 1 LEADER]: Becoming Leader. State: Replica: fb578c898e0f4bd5bbc541a4ab1f207f, State: Running, Role: LEADER
I20260929 14:42:23.478695 5071 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } }
I20260929 14:42:23.479233 5066 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:23.480252 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb578c898e0f4bd5bbc541a4ab1f207f. Latest consensus state: current_term: 1 leader_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } } }
I20260929 14:42:23.480360 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:23.480638 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb578c898e0f4bd5bbc541a4ab1f207f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41013 } } }
I20260929 14:42:23.480707 5080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:23.480722 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:23.483526 5080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:23.489269 5080 catalog_manager.cc:1347] Generated new cluster ID: 7ebb6c0a6968459dba13c5daa5839870
I20260929 14:42:23.489341 5080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:23.497171 5080 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:23.497453 5080 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:23.502965 5080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: Generated new TSK 0
I20260929 14:42:23.503605 5080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:23.515666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--builtin_ntp_servers=127.0.98.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:42:23.631374 5090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:23.631656 5090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:23.631697 5090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:23.635782 5090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:23.635857 5090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:23.635917 5090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:23.640957 5090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:23.642120 5090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:23.643589 5090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:23.654270 5095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:23.654275 5096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:23.654270 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:23.654692 5090 server_base.cc:1034] running on GCE node
I20260929 14:42:23.655117 5090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:23.655789 5090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:23.656944 5090 hybrid_clock.cc:648] HybridClock initialized: now 1790692943656915 us; error 48 us; skew 500 ppm
I20260929 14:42:23.658984 5090 webserver.cc:460] Webserver started at http://127.0.98.1:42317/ using document root <none> and password file <none>
I20260929 14:42:23.659565 5090 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:23.659628 5090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:23.659799 5090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:23.662119 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "6604e60d718d4514a216910196f3709d"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "adc02711c1f966696ff26256a64bfa75"
server_key_iv: "e4d7dc8c90b176cc96d2f25918c42420"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.662750 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6604e60d718d4514a216910196f3709d"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "adc02711c1f966696ff26256a64bfa75"
server_key_iv: "e4d7dc8c90b176cc96d2f25918c42420"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.666913 5090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:23.669843 5104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.671443 5090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:23.671586 5090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "6604e60d718d4514a216910196f3709d"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "adc02711c1f966696ff26256a64bfa75"
server_key_iv: "e4d7dc8c90b176cc96d2f25918c42420"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.671702 5090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:23.676448 5090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:23.677130 5090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:23.677284 5090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:23.677801 5090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:23.678721 5090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:23.678773 5090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.678825 5090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:23.678848 5090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.687104 5090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42257
I20260929 14:42:23.687125 5217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42257 every 8 connection(s)
I20260929 14:42:23.687994 5090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:23.691871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5090
I20260929 14:42:23.691987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:23.692243 392 external_mini_cluster.cc:1428] Setting key 87ea0d3bebd34c4345d8487c8c61d05f
I20260929 14:42:23.694211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--builtin_ntp_servers=127.0.98.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 14:42:23.694456 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.689885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59609 (local address 127.0.98.62:41013)
0929 14:42:23.690028 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.690032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.691235 (+ 1203us) server_negotiation.cc:407] Connection header received
0929 14:42:23.692603 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.692606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.692646 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.692717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:23.693955 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:23.693958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:23.693960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:23.693972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:23.693991 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:23.693999 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:23.694075 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.694266 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.694313 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:23.695547 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.690408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41013 (local address 127.0.98.1:59609)
0929 14:42:23.691084 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.691126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.692387 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.692950 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.692958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.693061 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:23.693631 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:23.693633 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:23.693699 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:23.693789 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:23.694078 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:23.694086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:23.694092 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.694268 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.694529 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260929 14:42:23.696779 5218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41013
I20260929 14:42:23.697111 5218 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:23.697516 5218 heartbeater.cc:511] Master 127.0.98.62:41013 requested a full tablet report, sending...
I20260929 14:42:23.699201 5029 ts_manager.cc:194] Registered new tserver with Master: 6604e60d718d4514a216910196f3709d (127.0.98.1:42257)
W20260929 14:42:23.805722 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:23.805977 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:23.806018 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:23.810070 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:23.810123 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:23.810179 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:23.815255 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:23.816298 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:23.817731 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:23.828109 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:23.828109 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:23.828114 5221 server_base.cc:1034] running on GCE node
W20260929 14:42:23.828135 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:23.828896 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:23.829600 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:23.830785 5221 hybrid_clock.cc:648] HybridClock initialized: now 1790692943830752 us; error 47 us; skew 500 ppm
I20260929 14:42:23.832979 5221 webserver.cc:460] Webserver started at http://127.0.98.2:41249/ using document root <none> and password file <none>
I20260929 14:42:23.833572 5221 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:23.833645 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:23.833851 5221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:23.835968 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "56ec6afb4d6b4c918711372d9ea56b49"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "99e8dc55cce26069e31e2ac25e05efa4"
server_key_iv: "99c7cebec704f8b0d7168c4d0f447ecf"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.836505 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56ec6afb4d6b4c918711372d9ea56b49"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "99e8dc55cce26069e31e2ac25e05efa4"
server_key_iv: "99c7cebec704f8b0d7168c4d0f447ecf"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.840212 5221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260929 14:42:23.842967 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.844311 5221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:23.844470 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "56ec6afb4d6b4c918711372d9ea56b49"
format_stamp: "Formatted at 2026-09-29 14:42:23 on dist-test-slave-g8hp"
server_key: "99e8dc55cce26069e31e2ac25e05efa4"
server_key_iv: "99c7cebec704f8b0d7168c4d0f447ecf"
server_key_version: "encryptionkey@0"
I20260929 14:42:23.844581 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:23.849363 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:23.850015 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:23.850167 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:23.850773 5221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:23.851681 5221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:23.851727 5221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.851778 5221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:23.851805 5221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:23.860658 5221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42489
I20260929 14:42:23.860682 5348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42489 every 8 connection(s)
I20260929 14:42:23.861631 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:23.868269 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.863672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44835 (local address 127.0.98.62:41013)
0929 14:42:23.863871 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:23.863876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:23.865074 (+ 1198us) server_negotiation.cc:407] Connection header received
0929 14:42:23.866437 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:23.866440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:23.866482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:23.866563 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:23.867780 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:23.867783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:23.867785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:23.867798 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:23.867819 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:23.867828 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:23.867928 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:23.868097 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 14:42:23.868146 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 14:42:23.869239 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:23.864118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41013 (local address 127.0.98.2:44835)
0929 14:42:23.864813 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:23.864855 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:23.866202 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:23.866796 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:23.866803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:23.866900 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:23.867471 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:23.867472 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:23.867516 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:23.867592 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:23.867919 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:23.867926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:23.867930 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:42:23.868088 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 14:42:23.868312 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
I20260929 14:42:23.870488 5349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41013
I20260929 14:42:23.870715 5349 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:23.870812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5221
I20260929 14:42:23.870981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:23.871150 5349 heartbeater.cc:511] Master 127.0.98.62:41013 requested a full tablet report, sending...
I20260929 14:42:23.871244 392 external_mini_cluster.cc:1428] Setting key b3c2f67fe6c84a43c93400e8742fc58e
I20260929 14:42:23.872236 5028 ts_manager.cc:194] Registered new tserver with Master: 56ec6afb4d6b4c918711372d9ea56b49 (127.0.98.2:42489)
I20260929 14:42:23.873404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--builtin_ntp_servers=127.0.98.20:44995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 14:42:23.987375 5352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:23.987677 5352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:23.987718 5352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:23.992089 5352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:23.992168 5352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:23.992228 5352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:23.997395 5352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:23.998543 5352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:23.999969 5352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:24.010227 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.010231 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.010227 5360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:24.010268 5352 server_base.cc:1034] running on GCE node
I20260929 14:42:24.010957 5352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:24.011649 5352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:24.012821 5352 hybrid_clock.cc:648] HybridClock initialized: now 1790692944012805 us; error 28 us; skew 500 ppm
I20260929 14:42:24.014923 5352 webserver.cc:460] Webserver started at http://127.0.98.3:38357/ using document root <none> and password file <none>
I20260929 14:42:24.015508 5352 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:24.015585 5352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:24.015779 5352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:24.017931 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "3f770610b5134b61bb4cfc9a01f186e2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "6fcdfd6ee9b0effcceaf9526c2e8c242"
server_key_iv: "9c20b51f9bd1aa19d2755aa14e677e6e"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.018460 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3f770610b5134b61bb4cfc9a01f186e2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "6fcdfd6ee9b0effcceaf9526c2e8c242"
server_key_iv: "9c20b51f9bd1aa19d2755aa14e677e6e"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.022250 5352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 14:42:24.024940 5366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:24.026238 5352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:24.026351 5352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "3f770610b5134b61bb4cfc9a01f186e2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "6fcdfd6ee9b0effcceaf9526c2e8c242"
server_key_iv: "9c20b51f9bd1aa19d2755aa14e677e6e"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.026461 5352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:24.031272 5352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:24.031908 5352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:24.032075 5352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:24.032655 5352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:24.033627 5352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:24.033675 5352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:24.033728 5352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:24.033757 5352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:24.042362 5352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38029
I20260929 14:42:24.042428 5479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38029 every 8 connection(s)
I20260929 14:42:24.043406 5352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:24.050137 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.045500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50853 (local address 127.0.98.62:41013)
0929 14:42:24.045654 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.045657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.046844 (+ 1187us) server_negotiation.cc:407] Connection header received
0929 14:42:24.048311 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.048315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.048363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.048442 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.049683 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.049687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.049690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.049703 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.049722 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.049730 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.049812 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.049942 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.049991 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 14:42:24.050206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5352
I20260929 14:42:24.050318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:24.050606 392 external_mini_cluster.cc:1428] Setting key 45e7d744c39ac5d6e485bf0ce8c2e868
I20260929 14:42:24.051286 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.045936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41013 (local address 127.0.98.3:50853)
0929 14:42:24.046645 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.046690 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.048091 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.048629 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.048636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.048752 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.049395 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.049396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.049440 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.049526 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.049800 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.049807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.049812 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.050091 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.050360 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260929 14:42:24.052441 5480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41013
I20260929 14:42:24.052731 5480 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:24.053215 5480 heartbeater.cc:511] Master 127.0.98.62:41013 requested a full tablet report, sending...
I20260929 14:42:24.054376 5029 ts_manager.cc:194] Registered new tserver with Master: 3f770610b5134b61bb4cfc9a01f186e2 (127.0.98.3:38029)
I20260929 14:42:24.065503 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:42:24.069339 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.067528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.98.62:41013)
0929 14:42:24.067665 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.067669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.067830 (+ 161us) server_negotiation.cc:407] Connection header received
0929 14:42:24.068058 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.068061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.068098 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.068169 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.068920 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.068923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.068925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.068936 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.068955 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.068963 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.069070 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.069156 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.069199 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:24.072496 5029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:24.076004 5029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:24.088460 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.085361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42489 (local address 127.0.0.1:43534)
0929 14:42:24.086155 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.086210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.086317 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.087010 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.087016 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.087095 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.087710 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.087712 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.087734 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.087803 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.088112 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.088121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.088126 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.088238 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.088308 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":416,"thread_start_us":107,"threads_started":1}
I20260929 14:42:24.088501 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.085592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.0.98.2:42489)
0929 14:42:24.086141 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.086151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.086416 (+ 265us) server_negotiation.cc:407] Connection header received
0929 14:42:24.086543 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.086549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.086740 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.086872 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.087942 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.087947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.087957 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.087967 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.088006 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.088030 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.088122 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.088269 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.088350 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":129,"threads_started":1}
I20260929 14:42:24.088990 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.085506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.1:42257)
0929 14:42:24.085815 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.085828 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.086301 (+ 473us) server_negotiation.cc:407] Connection header received
0929 14:42:24.086506 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.086513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.086716 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.086826 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.088347 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.088364 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.088376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.088388 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.088428 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.088460 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.088538 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.088712 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.088785 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260929 14:42:24.089308 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.085361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42257 (local address 127.0.0.1:43000)
0929 14:42:24.086155 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.086210 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.086377 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.087572 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.087575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.087590 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.088193 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.088195 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.088205 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.088229 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.088604 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.088607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.088610 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.089154 (+ 544us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.089189 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":98,"threads_started":1}
I20260929 14:42:24.089542 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.086385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.0.98.3:38029)
0929 14:42:24.086811 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.086826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.087070 (+ 244us) server_negotiation.cc:407] Connection header received
0929 14:42:24.087240 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.087246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.087419 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.087519 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.088900 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.088907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.088918 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.088931 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.088969 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.089006 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.089103 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.089309 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.089380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":159,"threads_started":1}
I20260929 14:42:24.089862 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.086144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38029 (local address 127.0.0.1:39870)
0929 14:42:24.086980 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.086994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.087123 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.088044 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.088047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.088059 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.088683 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.088685 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.088695 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.088722 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.089194 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.089197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.089200 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.089715 (+ 515us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.089748 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":83,"threads_started":1}
I20260929 14:42:24.091639 5283 tablet_service.cc:1467] Processing CreateTablet for tablet 73222c7fafd740868af9e9f5656e242b (DEFAULT_TABLE table=test-table [id=7593faa3081b4edd82837eec43362792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:24.092039 5152 tablet_service.cc:1467] Processing CreateTablet for tablet 73222c7fafd740868af9e9f5656e242b (DEFAULT_TABLE table=test-table [id=7593faa3081b4edd82837eec43362792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:24.092703 5283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73222c7fafd740868af9e9f5656e242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:24.092489 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 73222c7fafd740868af9e9f5656e242b (DEFAULT_TABLE table=test-table [id=7593faa3081b4edd82837eec43362792]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:24.093106 5152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73222c7fafd740868af9e9f5656e242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:24.093446 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73222c7fafd740868af9e9f5656e242b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:24.098829 5497 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Bootstrap starting.
I20260929 14:42:24.100597 5498 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Bootstrap starting.
I20260929 14:42:24.100597 5499 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Bootstrap starting.
I20260929 14:42:24.101439 5497 tablet_bootstrap.cc:654] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:24.102330 5497 log.cc:824] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:24.103139 5499 tablet_bootstrap.cc:654] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:24.103139 5498 tablet_bootstrap.cc:654] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:24.104012 5499 log.cc:824] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:24.104012 5498 log.cc:824] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:24.104162 5497 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: No bootstrap required, opened a new log
I20260929 14:42:24.104359 5497 ts_tablet_manager.cc:1397] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:24.106149 5498 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: No bootstrap required, opened a new log
I20260929 14:42:24.106149 5499 tablet_bootstrap.cc:492] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: No bootstrap required, opened a new log
I20260929 14:42:24.106348 5499 ts_tablet_manager.cc:1397] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 14:42:24.106348 5498 ts_tablet_manager.cc:1397] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:42:24.108237 5497 raft_consensus.cc:348] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.108481 5497 raft_consensus.cc:374] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:24.108553 5497 raft_consensus.cc:729] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ec6afb4d6b4c918711372d9ea56b49, State: Initialized, Role: FOLLOWER
I20260929 14:42:24.109066 5497 consensus_queue.cc:260] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.109741 5349 heartbeater.cc:503] Master 127.0.98.62:41013 was elected leader, sending a full tablet report...
I20260929 14:42:24.110158 5497 ts_tablet_manager.cc:1428] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:42:24.110868 5498 raft_consensus.cc:348] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.110868 5499 raft_consensus.cc:348] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.111166 5499 raft_consensus.cc:374] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:24.111166 5498 raft_consensus.cc:374] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:24.111235 5498 raft_consensus.cc:729] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f770610b5134b61bb4cfc9a01f186e2, State: Initialized, Role: FOLLOWER
I20260929 14:42:24.111258 5499 raft_consensus.cc:729] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6604e60d718d4514a216910196f3709d, State: Initialized, Role: FOLLOWER
I20260929 14:42:24.111750 5499 consensus_queue.cc:260] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.111743 5498 consensus_queue.cc:260] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.112686 5480 heartbeater.cc:503] Master 127.0.98.62:41013 was elected leader, sending a full tablet report...
I20260929 14:42:24.112847 5498 ts_tablet_manager.cc:1428] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260929 14:42:24.112847 5499 ts_tablet_manager.cc:1428] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20260929 14:42:24.113047 5350 tablet.cc:2369] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:24.113212 5218 heartbeater.cc:503] Master 127.0.98.62:41013 was elected leader, sending a full tablet report...
I20260929 14:42:24.154769 5504 raft_consensus.cc:482] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:24.155061 5504 raft_consensus.cc:504] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.156201 5504 leader_election.cc:288] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56ec6afb4d6b4c918711372d9ea56b49 (127.0.98.2:42489), 3f770610b5134b61bb4cfc9a01f186e2 (127.0.98.3:38029)
I20260929 14:42:24.158147 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.156478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42489 (local address 127.0.98.1:41223)
0929 14:42:24.156591 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.156607 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.156695 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.157100 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.157103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.157116 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.157609 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.157610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.157620 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.157643 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.157890 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.157893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.157897 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.157994 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.158028 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 14:42:24.158154 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.156586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41223 (local address 127.0.98.2:42489)
0929 14:42:24.156714 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.156718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.156736 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:42:24.156790 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.156792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.156830 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.157072 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.157743 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.157747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.157750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.157762 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.157796 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.157808 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.157890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.157988 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.158025 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:24.158562 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.156743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38029 (local address 127.0.98.1:54373)
0929 14:42:24.157108 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.157121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.157202 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.157438 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.157441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.157456 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:24.157947 (+ 491us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:24.157949 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:24.157960 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:24.157983 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:24.158336 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:24.158340 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:24.158343 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 14:42:24.158424 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 14:42:24.158458 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260929 14:42:24.158602 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.156912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54373 (local address 127.0.98.3:38029)
0929 14:42:24.157033 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.157037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.157200 (+ 163us) server_negotiation.cc:407] Connection header received
0929 14:42:24.157262 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.157264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.157300 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.157361 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.158099 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.158103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.158106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.158116 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.158136 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.158146 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.158233 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.158447 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.158489 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:24.158911 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73222c7fafd740868af9e9f5656e242b" candidate_uuid: "6604e60d718d4514a216910196f3709d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ec6afb4d6b4c918711372d9ea56b49" is_pre_election: true
I20260929 14:42:24.159193 5303 raft_consensus.cc:2454] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6604e60d718d4514a216910196f3709d in term 0.
I20260929 14:42:24.159658 5106 leader_election.cc:302] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ec6afb4d6b4c918711372d9ea56b49, 6604e60d718d4514a216910196f3709d; no voters:
I20260929 14:42:24.159948 5504 raft_consensus.cc:2781] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:24.160032 5504 raft_consensus.cc:482] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:24.159989 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73222c7fafd740868af9e9f5656e242b" candidate_uuid: "6604e60d718d4514a216910196f3709d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f770610b5134b61bb4cfc9a01f186e2" is_pre_election: true
I20260929 14:42:24.160082 5504 raft_consensus.cc:3037] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:24.160241 5434 raft_consensus.cc:2454] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6604e60d718d4514a216910196f3709d in term 0.
I20260929 14:42:24.161265 5504 raft_consensus.cc:504] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.161581 5504 leader_election.cc:288] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [CANDIDATE]: Term 1 election: Requested vote from peers 56ec6afb4d6b4c918711372d9ea56b49 (127.0.98.2:42489), 3f770610b5134b61bb4cfc9a01f186e2 (127.0.98.3:38029)
I20260929 14:42:24.161917 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73222c7fafd740868af9e9f5656e242b" candidate_uuid: "6604e60d718d4514a216910196f3709d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56ec6afb4d6b4c918711372d9ea56b49"
I20260929 14:42:24.162055 5303 raft_consensus.cc:3037] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:24.162107 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73222c7fafd740868af9e9f5656e242b" candidate_uuid: "6604e60d718d4514a216910196f3709d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f770610b5134b61bb4cfc9a01f186e2"
I20260929 14:42:24.162232 5434 raft_consensus.cc:3037] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:24.163147 5303 raft_consensus.cc:2454] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6604e60d718d4514a216910196f3709d in term 1.
I20260929 14:42:24.163239 5434 raft_consensus.cc:2454] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6604e60d718d4514a216910196f3709d in term 1.
I20260929 14:42:24.163442 5106 leader_election.cc:302] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56ec6afb4d6b4c918711372d9ea56b49, 6604e60d718d4514a216910196f3709d; no voters:
I20260929 14:42:24.163606 5504 raft_consensus.cc:2781] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:24.163838 5504 raft_consensus.cc:686] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 1 LEADER]: Becoming Leader. State: Replica: 6604e60d718d4514a216910196f3709d, State: Running, Role: LEADER
I20260929 14:42:24.164198 5504 consensus_queue.cc:237] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } }
I20260929 14:42:24.167676 5029 catalog_manager.cc:5495] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6604e60d718d4514a216910196f3709d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6604e60d718d4514a216910196f3709d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6604e60d718d4514a216910196f3709d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 } health_report { overall_health: UNKNOWN } } }
W20260929 14:42:24.189136 5219 tablet.cc:2369] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:24.189522 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.187686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.0.98.1:42257)
0929 14:42:24.187814 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.187818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.187831 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:24.188103 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.188106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.188173 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.188252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.189126 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.189129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.189131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.189143 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.189162 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.189170 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.189250 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.189321 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.189361 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:24.195283 5303 raft_consensus.cc:1261] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 1 FOLLOWER]: Refusing update from remote peer 6604e60d718d4514a216910196f3709d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:24.195343 5434 raft_consensus.cc:1261] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 1 FOLLOWER]: Refusing update from remote peer 6604e60d718d4514a216910196f3709d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:24.195981 5507 consensus_queue.cc:1035] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56ec6afb4d6b4c918711372d9ea56b49" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:24.196164 5504 consensus_queue.cc:1035] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f770610b5134b61bb4cfc9a01f186e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:24.202827 5510 mvcc.cc:205] Tried to move back new op lower bound from 7334678299416854528 to 7334678299299708928. Current Snapshot: MvccSnapshot[applied={T|T < 7334678299416854528}]
I20260929 14:42:24.204227 5512 mvcc.cc:205] Tried to move back new op lower bound from 7334678299416854528 to 7334678299299708928. Current Snapshot: MvccSnapshot[applied={T|T < 7334678299416854528}]
I20260929 14:42:24.212276 5029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:24.212481 5029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:24.215152 5029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb578c898e0f4bd5bbc541a4ab1f207f: Sending DeleteTablet for 3 replicas of tablet 73222c7fafd740868af9e9f5656e242b
I20260929 14:42:24.215977 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 73222c7fafd740868af9e9f5656e242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:24 UTC) from {username='slave'} at 127.0.0.1:43000
I20260929 14:42:24.216177 5283 tablet_service.cc:1514] Processing DeleteTablet for tablet 73222c7fafd740868af9e9f5656e242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:24 UTC) from {username='slave'} at 127.0.0.1:43534
I20260929 14:42:24.216554 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 73222c7fafd740868af9e9f5656e242b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:24 UTC) from {username='slave'} at 127.0.0.1:39870
I20260929 14:42:24.216678 5521 tablet_replica.cc:330] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: stopping tablet replica
I20260929 14:42:24.217026 5522 tablet_replica.cc:330] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: stopping tablet replica
I20260929 14:42:24.217100 5521 raft_consensus.cc:2229] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:24.217365 5522 raft_consensus.cc:2229] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:24.217460 5520 tablet_replica.cc:330] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d: stopping tablet replica
I20260929 14:42:24.217598 5521 raft_consensus.cc:2258] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:24.217723 5522 raft_consensus.cc:2258] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:24.217761 5520 raft_consensus.cc:2229] T 73222c7fafd740868af9e9f5656e242b P 6604e60d718d4514a216910196f3709d [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:24.217821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5090
I20260929 14:42:24.219799 5522 ts_tablet_manager.cc:1905] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:24.220975 5521 ts_tablet_manager.cc:1905] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:24.222127 5522 ts_tablet_manager.cc:1918] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:24.222223 5522 log.cc:1196] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/73222c7fafd740868af9e9f5656e242b
I20260929 14:42:24.222491 5522 ts_tablet_manager.cc:1939] T 73222c7fafd740868af9e9f5656e242b P 3f770610b5134b61bb4cfc9a01f186e2: Deleting consensus metadata
I20260929 14:42:24.223423 5521 ts_tablet_manager.cc:1918] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:24.223559 5521 log.cc:1196] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/73222c7fafd740868af9e9f5656e242b
I20260929 14:42:24.223661 5015 catalog_manager.cc:4841] TS 3f770610b5134b61bb4cfc9a01f186e2 (127.0.98.3:38029): tablet 73222c7fafd740868af9e9f5656e242b (table test-table [id=7593faa3081b4edd82837eec43362792]) successfully deleted
I20260929 14:42:24.223832 5521 ts_tablet_manager.cc:1939] T 73222c7fafd740868af9e9f5656e242b P 56ec6afb4d6b4c918711372d9ea56b49: Deleting consensus metadata
I20260929 14:42:24.225025 5015 catalog_manager.cc:4841] TS 56ec6afb4d6b4c918711372d9ea56b49 (127.0.98.2:42489): tablet 73222c7fafd740868af9e9f5656e242b (table test-table [id=7593faa3081b4edd82837eec43362792]) successfully deleted
W20260929 14:42:24.225389 5014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42257 (error 108)
I20260929 14:42:24.225605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5221
I20260929 14:42:24.226890 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.226616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42257 (local address 127.0.0.1:43014)
0929 14:42:24.226745 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.226815 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 14:42:24.227147 5014 catalog_manager.cc:4584] TS 6604e60d718d4514a216910196f3709d (127.0.98.1:42257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73222c7fafd740868af9e9f5656e242b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42257: connect: Connection refused (error 111)
I20260929 14:42:24.230792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5352
I20260929 14:42:24.235451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 4999
2026-09-29T14:42:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (975 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 14:42:24.255369 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:42:24.257004 392 webserver.cc:460] Webserver started at https://127.0.0.1:42647/ using document root <none> and password file <none>
I20260929 14:42:24.257181 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:42:24.257436 392 webserver.cc:460] Webserver started at http://127.0.0.1:45399/ using document root <none> and password file <none>
2026-09-29T14:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:24Z Disabled control of system clock
I20260929 14:42:24.266736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35223
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/testchainca.cert with env {}
W20260929 14:42:24.377861 5529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:24.378156 5529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:24.378196 5529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:24.382283 5529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:24.382352 5529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:24.382367 5529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:24.382383 5529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:24.382397 5529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:42:24.382413 5529 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:42:24.382424 5529 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42647/jwks.json
I20260929 14:42:24.388073 5529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:24.389276 5529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:24.390291 5529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:24.399188 5537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.399207 5534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.399197 5535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:24.399430 5529 server_base.cc:1034] running on GCE node
I20260929 14:42:24.400182 5529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:24.401482 5529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:24.402660 5529 hybrid_clock.cc:648] HybridClock initialized: now 1790692944402642 us; error 31 us; skew 500 ppm
I20260929 14:42:24.404729 5529 webserver.cc:460] Webserver started at http://127.0.98.62:35461/ using document root <none> and password file <none>
I20260929 14:42:24.405335 5529 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:24.405401 5529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:24.405599 5529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:24.407662 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "2af80d026ccf4dc6bcb411d696e5e4ba"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "786e33d1f723a91605d4416018aeb6e4"
server_key_iv: "701bc061c95581381dcd0cb793edd441"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.408238 5529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2af80d026ccf4dc6bcb411d696e5e4ba"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "786e33d1f723a91605d4416018aeb6e4"
server_key_iv: "701bc061c95581381dcd0cb793edd441"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.412290 5529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:24.414970 5543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:24.416386 5529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:24.416508 5529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "2af80d026ccf4dc6bcb411d696e5e4ba"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "786e33d1f723a91605d4416018aeb6e4"
server_key_iv: "701bc061c95581381dcd0cb793edd441"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.416638 5529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:24.437129 5529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:24.437893 5529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:24.438079 5529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:24.445207 5529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35223
I20260929 14:42:24.445230 5595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35223 every 8 connection(s)
I20260929 14:42:24.446178 5529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:24.449456 5596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:24.453087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5529
I20260929 14:42:24.453213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:24.453486 392 external_mini_cluster.cc:1428] Setting key 524419fbdd09833c2ffe6b4a32849cce
I20260929 14:42:24.455651 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba: Bootstrap starting.
I20260929 14:42:24.457988 5596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:24.458734 5596 log.cc:824] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:24.460557 5596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba: No bootstrap required, opened a new log
I20260929 14:42:24.462692 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.454956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.98.62:35223)
0929 14:42:24.455657 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.455672 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.455709 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:42:24.456410 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.456429 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.456832 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.457182 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.457635 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.458646 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.459556 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.460019 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.460156 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.460981 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.460987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.461001 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.461397 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.461457 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.461476 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.461559 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.461717 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.461973 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":126,"threads_started":1}
I20260929 14:42:24.464242 5596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260929 14:42:24.464509 5596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:24.464569 5596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2af80d026ccf4dc6bcb411d696e5e4ba, State: Initialized, Role: FOLLOWER
I20260929 14:42:24.465046 5596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260929 14:42:24.465200 5596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:24.465274 5596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:24.465372 5596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:24.465442 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:42:24.466289 5596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260929 14:42:24.466609 5596 leader_election.cc:302] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2af80d026ccf4dc6bcb411d696e5e4ba; no voters:
I20260929 14:42:24.466907 5596 leader_election.cc:288] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:24.467007 5601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:24.467172 5601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [term 1 LEADER]: Becoming Leader. State: Replica: 2af80d026ccf4dc6bcb411d696e5e4ba, State: Running, Role: LEADER
I20260929 14:42:24.467510 5601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } }
I20260929 14:42:24.468214 5596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:24.469321 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2af80d026ccf4dc6bcb411d696e5e4ba. Latest consensus state: current_term: 1 leader_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260929 14:42:24.469449 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:24.469691 5602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2af80d026ccf4dc6bcb411d696e5e4ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35223 } } }
I20260929 14:42:24.469801 5602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:24.469898 5610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:24.472946 5610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:24.478855 5610 catalog_manager.cc:1347] Generated new cluster ID: 155a437ee04745c3a8808dc7e162cd57
I20260929 14:42:24.478931 5610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:24.498421 5610 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:24.499383 5610 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:24.505326 5610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2af80d026ccf4dc6bcb411d696e5e4ba: Generated new TSK 0
I20260929 14:42:24.506052 5610 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:42:24.552040 5626 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 14:42:24.552150 5626 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:42:24.552709 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.528343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.98.62:35223)
0929 14:42:24.528493 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.528498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.528740 (+ 242us) server_negotiation.cc:407] Connection header received
0929 14:42:24.529021 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.529024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.529076 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.529160 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:42:24.529470 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.530103 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.531374 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.531605 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.531688 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.532060 (+ 372us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:42:24.552263 (+ 20203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:42:24.552565 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 14:42:24.552809 5599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43846: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 14:42:24.553898 5625 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:24.528233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35223 (local address 127.0.0.1:43846)
0929 14:42:24.528622 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.528637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.528793 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.529194 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.529197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.529205 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:42:24.529329 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:24.529335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.530229 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.530233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:24.531238 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:24.531248 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.531694 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.531716 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:24.531803 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.531860 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:42:24.552507 (+ 20647us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 14:42:24.552723 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 14:42:24.554563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5529
2026-09-29T14:42:24Z chronyd exiting
I20260929 14:42:24.591219 392 test_util.cc:175] -----------------------------------------------
I20260929 14:42:24.591310 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790692915364895-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (337 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 14:42:24.592399 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:42:24.593295 392 webserver.cc:460] Webserver started at https://127.0.0.1:35799/ using document root <none> and password file <none>
I20260929 14:42:24.593389 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:42:24.593636 392 webserver.cc:460] Webserver started at http://127.0.0.1:41267/ using document root <none> and password file <none>
2026-09-29T14:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:24Z Disabled control of system clock
I20260929 14:42:24.602654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35799/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/testchainca.cert with env {}
W20260929 14:42:24.713599 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:24.713888 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:24.713929 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:24.718003 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:24.718063 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:24.718078 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:24.718094 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:24.718108 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:42:24.718122 5632 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:42:24.718137 5632 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35799/jwks.json
I20260929 14:42:24.723628 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35799/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:24.724735 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:24.725795 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:24.734298 5638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.734308 5637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:24.734298 5640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:24.734659 5632 server_base.cc:1034] running on GCE node
I20260929 14:42:24.735302 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:24.736446 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:24.737622 5632 hybrid_clock.cc:648] HybridClock initialized: now 1790692944737552 us; error 87 us; skew 500 ppm
I20260929 14:42:24.739669 5632 webserver.cc:460] Webserver started at http://127.0.98.62:34569/ using document root <none> and password file <none>
I20260929 14:42:24.740209 5632 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:24.740268 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:24.740439 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:24.742563 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "b027d72cbb2d40c1ac5568c3977185c2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "5502ca45eb8024fa331fd515369df238"
server_key_iv: "de1a8199d5b72be5fa3434d9047a3f0d"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.743095 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "b027d72cbb2d40c1ac5568c3977185c2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "5502ca45eb8024fa331fd515369df238"
server_key_iv: "de1a8199d5b72be5fa3434d9047a3f0d"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.746868 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:24.749408 5646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:24.750686 5632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:24.750801 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "b027d72cbb2d40c1ac5568c3977185c2"
format_stamp: "Formatted at 2026-09-29 14:42:24 on dist-test-slave-g8hp"
server_key: "5502ca45eb8024fa331fd515369df238"
server_key_iv: "de1a8199d5b72be5fa3434d9047a3f0d"
server_key_version: "encryptionkey@0"
I20260929 14:42:24.750914 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:24.774107 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:24.774885 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:24.775074 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:24.781853 5632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45269
I20260929 14:42:24.781875 5698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45269 every 8 connection(s)
I20260929 14:42:24.782809 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:24.785964 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:24.788901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5632
I20260929 14:42:24.789026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:24.789300 392 external_mini_cluster.cc:1428] Setting key 7f28e06fc1aa0ed01935ff3f1cb7d812
I20260929 14:42:24.792300 5699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: Bootstrap starting.
I20260929 14:42:24.794719 5699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:24.795446 5699 log.cc:824] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:24.797645 5699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: No bootstrap required, opened a new log
I20260929 14:42:24.799341 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.790730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.98.62:45269)
0929 14:42:24.791427 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.791443 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.791479 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:24.792120 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.792137 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.792588 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.792960 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:24.793429 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.794491 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.795411 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.796127 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.796287 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.797330 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:24.797335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:24.797350 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:24.797767 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:24.797849 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:24.797879 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:24.797972 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:24.798170 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 14:42:24.798478 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260929 14:42:24.801414 5699 raft_consensus.cc:348] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260929 14:42:24.801679 5699 raft_consensus.cc:374] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:24.801740 5699 raft_consensus.cc:729] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b027d72cbb2d40c1ac5568c3977185c2, State: Initialized, Role: FOLLOWER
I20260929 14:42:24.802155 5699 consensus_queue.cc:260] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260929 14:42:24.802285 5699 raft_consensus.cc:388] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:24.802351 5699 raft_consensus.cc:482] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:24.802409 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:42:24.802456 5699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:24.803592 5699 raft_consensus.cc:504] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260929 14:42:24.803931 5699 leader_election.cc:302] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b027d72cbb2d40c1ac5568c3977185c2; no voters:
I20260929 14:42:24.804234 5699 leader_election.cc:288] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:24.804334 5704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:24.804584 5704 raft_consensus.cc:686] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [term 1 LEADER]: Becoming Leader. State: Replica: b027d72cbb2d40c1ac5568c3977185c2, State: Running, Role: LEADER
I20260929 14:42:24.804957 5704 consensus_queue.cc:237] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } }
I20260929 14:42:24.805569 5699 sys_catalog.cc:564] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:24.806613 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b027d72cbb2d40c1ac5568c3977185c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260929 14:42:24.806754 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:24.806962 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b027d72cbb2d40c1ac5568c3977185c2. Latest consensus state: current_term: 1 leader_uuid: "b027d72cbb2d40c1ac5568c3977185c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b027d72cbb2d40c1ac5568c3977185c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45269 } } }
I20260929 14:42:24.807040 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2 [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:24.810550 5719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:24.810616 5719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:24.810686 5713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:24.811381 5713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:24.817337 5713 catalog_manager.cc:1347] Generated new cluster ID: de905a77459043ae99239b19395a96b3
I20260929 14:42:24.817415 5713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:24.832643 5713 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:24.833810 5713 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:24.848353 5713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b027d72cbb2d40c1ac5568c3977185c2: Generated new TSK 0
I20260929 14:42:24.849282 5713 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 14:42:24.877632 5729 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 14:42:24.877727 5729 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 14:42:24.878500 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:24.861948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.0.98.62:45269)
0929 14:42:24.862079 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:24.862084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:24.862208 (+ 124us) server_negotiation.cc:407] Connection header received
0929 14:42:24.862384 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:24.862388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:24.862441 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:24.862538 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:42:24.862766 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.863327 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.864185 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.864414 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.864499 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.864790 (+ 291us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:42:24.878037 (+ 13247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:42:24.878339 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 14:42:24.878579 5702 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 14:42:24.878639 5728 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:24.861820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45269 (local address 127.0.0.1:57754)
0929 14:42:24.862115 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:24.862128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:24.862246 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:24.862545 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:24.862548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:24.862556 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:42:24.862667 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:24.862672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.863435 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.863439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:24.864073 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:24.864081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:24.864499 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:24.864521 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:24.864603 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:24.864626 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:42:24.878294 (+ 13668us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 14:42:24.878473 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35799/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":109,"threads_started":1}
I20260929 14:42:24.879446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5632
2026-09-29T14:42:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 14:42:24.917740 392 mini_oidc.cc:152] Starting JWKS server
I20260929 14:42:24.918596 392 webserver.cc:460] Webserver started at https://127.0.0.1:41645/ using document root <none> and password file <none>
I20260929 14:42:24.918681 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 14:42:24.918903 392 webserver.cc:460] Webserver started at http://127.0.0.1:33899/ using document root <none> and password file <none>
2026-09-29T14:42:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:24Z Disabled control of system clock
I20260929 14:42:24.927675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38661
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:42:25.040083 5735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:25.040376 5735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:25.040416 5735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:25.044507 5735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:25.044584 5735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:25.044598 5735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:25.044615 5735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:25.044628 5735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:42:25.044643 5735 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:42:25.044655 5735 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41645/jwks.json
I20260929 14:42:25.049783 5735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:25.050876 5735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:25.052239 5735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:25.060948 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:25.060962 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:25.060956 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:25.061043 5735 server_base.cc:1034] running on GCE node
I20260929 14:42:25.061911 5735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:25.063073 5735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:25.064249 5735 hybrid_clock.cc:648] HybridClock initialized: now 1790692945064221 us; error 44 us; skew 500 ppm
I20260929 14:42:25.066511 5735 webserver.cc:460] Webserver started at http://127.0.98.62:34433/ using document root <none> and password file <none>
I20260929 14:42:25.067224 5735 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:25.067294 5735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:25.067502 5735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:25.069820 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "24e0920da3aa445c821b74144b241783"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "25f90772e8df07f38c613c2f5e7b14c3"
server_key_iv: "116634492b2f587b4818f7721dafd329"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.070432 5735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "24e0920da3aa445c821b74144b241783"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "25f90772e8df07f38c613c2f5e7b14c3"
server_key_iv: "116634492b2f587b4818f7721dafd329"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.074505 5735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:25.077278 5749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:25.078631 5735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:25.078750 5735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "24e0920da3aa445c821b74144b241783"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "25f90772e8df07f38c613c2f5e7b14c3"
server_key_iv: "116634492b2f587b4818f7721dafd329"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.078868 5735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:25.108502 5735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:25.109336 5735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:25.109519 5735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:25.116436 5735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38661
I20260929 14:42:25.116459 5801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38661 every 8 connection(s)
I20260929 14:42:25.117689 5735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:25.121187 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:25.124017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5735
I20260929 14:42:25.124158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:25.124437 392 external_mini_cluster.cc:1428] Setting key 0fd32d58c2f52dd9a64b160574513ee9
I20260929 14:42:25.127717 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: Bootstrap starting.
I20260929 14:42:25.130129 5802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:25.130884 5802 log.cc:824] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:25.132848 5802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: No bootstrap required, opened a new log
I20260929 14:42:25.134287 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:25.125941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46082 (local address 127.0.98.62:38661)
0929 14:42:25.126651 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:25.126667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:25.126712 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:42:25.127381 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:25.127400 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:25.127839 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:25.128191 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:25.128684 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.129779 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.130712 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.131239 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.131389 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.132194 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:25.132207 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:25.132221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:25.132694 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:25.132902 (+ 208us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:25.132941 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:25.133052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:25.133187 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 14:42:25.133471 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260929 14:42:25.136056 5802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } }
I20260929 14:42:25.136498 5802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:25.136579 5802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e0920da3aa445c821b74144b241783, State: Initialized, Role: FOLLOWER
I20260929 14:42:25.137071 5802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } }
I20260929 14:42:25.137248 5802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:25.137326 5802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:25.137413 5802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:25.137676 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:42:25.138453 5802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } }
I20260929 14:42:25.138808 5802 leader_election.cc:302] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24e0920da3aa445c821b74144b241783; no voters:
I20260929 14:42:25.139142 5802 leader_election.cc:288] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:25.139252 5807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:25.139430 5807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [term 1 LEADER]: Becoming Leader. State: Replica: 24e0920da3aa445c821b74144b241783, State: Running, Role: LEADER
I20260929 14:42:25.139756 5807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } }
I20260929 14:42:25.140466 5802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:25.141403 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24e0920da3aa445c821b74144b241783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } }
I20260929 14:42:25.141558 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:25.141805 5808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24e0920da3aa445c821b74144b241783. Latest consensus state: current_term: 1 leader_uuid: "24e0920da3aa445c821b74144b241783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e0920da3aa445c821b74144b241783" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } }
I20260929 14:42:25.141891 5808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783 [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:25.145552 5822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:25.145655 5822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:25.145773 5816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:25.146584 5816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:25.151535 5816 catalog_manager.cc:1347] Generated new cluster ID: ef546ccd18554c908e322545a4b4e5b9
I20260929 14:42:25.151607 5816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:25.170500 5816 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:25.171821 5816 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:25.181542 5816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24e0920da3aa445c821b74144b241783: Generated new TSK 0
I20260929 14:42:25.182461 5816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:25.198850 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:25.193698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.98.62:38661)
0929 14:42:25.193943 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:25.193951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:25.193968 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:42:25.194155 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:25.194158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:25.194206 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:25.194305 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:42:25.194545 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.195188 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.196494 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.196725 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.196814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.197177 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:42:25.198479 (+ 1302us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:42:25.198714 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 14:42:25.198930 5805 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46096: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 14:42:25.201797 5831 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:25.193571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38661 (local address 127.0.0.1:46096)
0929 14:42:25.193893 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:25.193908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:25.194024 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:25.194317 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:25.194321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:25.194328 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:42:25.194437 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:25.194444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.195298 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.195301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:25.196315 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:25.196327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.196822 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.196850 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:25.196970 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.196999 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:42:25.198653 (+ 1654us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 14:42:25.201606 (+ 2953us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2621,"thread_start_us":119,"threads_started":1}
I20260929 14:42:25.202220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5735
E20260929 14:42:25.211014 5832 webserver.cc:533] Webserver: sslize failed:
E20260929 14:42:25.211113 5832 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T14:42:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T14:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:25Z Disabled control of system clock
I20260929 14:42:25.248740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44575
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44575
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/testchainca.cert with env {}
W20260929 14:42:25.360405 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:25.360697 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:25.360736 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:25.364837 5836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:25.364930 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:25.364945 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:25.364965 5836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:25.364980 5836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 14:42:25.364995 5836 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 14:42:25.365007 5836 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 14:42:25.370697 5836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:25.371789 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:25.372780 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:25.382130 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:25.382185 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:25.382344 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:25.382480 5836 server_base.cc:1034] running on GCE node
I20260929 14:42:25.383157 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:25.384372 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:25.385536 5836 hybrid_clock.cc:648] HybridClock initialized: now 1790692945385473 us; error 81 us; skew 500 ppm
I20260929 14:42:25.387598 5836 webserver.cc:460] Webserver started at http://127.0.98.62:44909/ using document root <none> and password file <none>
I20260929 14:42:25.388206 5836 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:25.388281 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:25.388480 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:25.390691 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "5834be8b996547fabe8f3a20c7aa6d37"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "a1d58feb02b0de4235b80c6e2e72c72e"
server_key_iv: "c2c2c687b71081265b83248f7752a127"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.391296 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "5834be8b996547fabe8f3a20c7aa6d37"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "a1d58feb02b0de4235b80c6e2e72c72e"
server_key_iv: "c2c2c687b71081265b83248f7752a127"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.395279 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:25.397943 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:25.399253 5836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:25.399382 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "5834be8b996547fabe8f3a20c7aa6d37"
format_stamp: "Formatted at 2026-09-29 14:42:25 on dist-test-slave-g8hp"
server_key: "a1d58feb02b0de4235b80c6e2e72c72e"
server_key_iv: "c2c2c687b71081265b83248f7752a127"
server_key_version: "encryptionkey@0"
I20260929 14:42:25.399493 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:25.418818 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:25.419557 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:25.419742 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:25.426849 5836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44575
I20260929 14:42:25.426882 5902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44575 every 8 connection(s)
I20260929 14:42:25.427835 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:25.431049 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:25.434667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5836
I20260929 14:42:25.434788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:25.435041 392 external_mini_cluster.cc:1428] Setting key 8bffa5c1289af4681f9226440458ed04
I20260929 14:42:25.437932 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37: Bootstrap starting.
I20260929 14:42:25.440526 5903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:25.441556 5903 log.cc:824] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:25.443988 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37: No bootstrap required, opened a new log
I20260929 14:42:25.445698 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:25.436457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.0.98.62:44575)
0929 14:42:25.437251 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:25.437267 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:25.437314 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:42:25.437983 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:25.438006 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:25.438450 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:25.438751 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:25.439222 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.440254 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.441603 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.442321 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.442485 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.443606 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:25.443611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:25.443626 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:25.444192 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:25.444261 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:25.444290 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:25.444374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:25.444522 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 14:42:25.444953 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":114,"threads_started":1}
I20260929 14:42:25.447727 5903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260929 14:42:25.448011 5903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:25.448074 5903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5834be8b996547fabe8f3a20c7aa6d37, State: Initialized, Role: FOLLOWER
I20260929 14:42:25.448487 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:42:25.448516 5903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260929 14:42:25.448688 5903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:25.448750 5903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:25.448848 5903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:25.449923 5903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260929 14:42:25.450295 5903 leader_election.cc:302] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5834be8b996547fabe8f3a20c7aa6d37; no voters:
I20260929 14:42:25.450618 5903 leader_election.cc:288] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:25.450719 5908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:25.450956 5908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [term 1 LEADER]: Becoming Leader. State: Replica: 5834be8b996547fabe8f3a20c7aa6d37, State: Running, Role: LEADER
I20260929 14:42:25.451251 5908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } }
I20260929 14:42:25.451819 5903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:25.452937 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5834be8b996547fabe8f3a20c7aa6d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } } }
I20260929 14:42:25.453059 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:25.453013 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5834be8b996547fabe8f3a20c7aa6d37. Latest consensus state: current_term: 1 leader_uuid: "5834be8b996547fabe8f3a20c7aa6d37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5834be8b996547fabe8f3a20c7aa6d37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44575 } } }
I20260929 14:42:25.453101 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:25.453395 5917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:25.456321 5917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:25.461802 5917 catalog_manager.cc:1347] Generated new cluster ID: 18be66b6268e434c9e1480e368a75bdd
I20260929 14:42:25.461875 5917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:25.479483 5917 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:25.480788 5917 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:25.486825 5917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5834be8b996547fabe8f3a20c7aa6d37: Generated new TSK 0
I20260929 14:42:25.487695 5917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:25.561393 5906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:25.494830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.0.98.62:44575)
0929 14:42:25.494950 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:25.494961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:25.495175 (+ 214us) server_negotiation.cc:407] Connection header received
0929 14:42:25.495322 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:25.495324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:25.495376 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:25.495467 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0929 14:42:25.495724 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.496330 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.497652 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.497896 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.497991 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.498346 (+ 355us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 14:42:25.560989 (+ 62643us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:42:25.561272 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41604: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 14:42:25.561473 5906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41604: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 14:42:25.561694 5932 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:25.494710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44575 (local address 127.0.0.1:41604)
0929 14:42:25.495057 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:25.495073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:25.495199 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:25.495489 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:25.495492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:25.495500 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0929 14:42:25.495617 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:25.495623 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.496457 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.496461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:25.497510 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:25.497522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:25.498001 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:25.498027 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:25.498119 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:25.498149 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 14:42:25.561250 (+ 63101us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 14:42:25.561483 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44575: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":101,"threads_started":1}
I20260929 14:42:25.562476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5836
2026-09-29T14:42:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (339 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (123 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:25 dist-test-slave-g8hp krb5kdc[5944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:25 dist-test-slave-g8hp krb5kdc[5944](info): set up 2 sockets
Sep 29 14:42:25 dist-test-slave-g8hp krb5kdc[5944](info): commencing operation
krb5kdc: starting...
W20260929 14:42:26.765039 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:26 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:26.886215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:26.998337 5957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:26.998644 5957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:26.998687 5957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:27.002835 5957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:27.002913 5957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:27.002933 5957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:27.002949 5957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:27.002964 5957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:27.008757 5957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:27.009892 5957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:27.010886 5957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:27.019500 5962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.019616 5963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.019532 5965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:27.019670 5957 server_base.cc:1034] running on GCE node
I20260929 14:42:27.020465 5957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:27.021701 5957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:27.022879 5957 hybrid_clock.cc:648] HybridClock initialized: now 1790692947022861 us; error 36 us; skew 500 ppm
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.025756 5957 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:27.026671 5957 webserver.cc:460] Webserver started at http://127.0.98.62:43849/ using document root <none> and password file <none>
I20260929 14:42:27.027210 5957 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:27.027271 5957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:27.027432 5957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:27.029578 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "dc123d4493f844ceacf13590d301db8c"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8121d1604a29b9a6bcfff1ee7aa1656a"
server_key_iv: "d957616b75a7ad994860a51995bee36e"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.030110 5957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc123d4493f844ceacf13590d301db8c"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8121d1604a29b9a6bcfff1ee7aa1656a"
server_key_iv: "d957616b75a7ad994860a51995bee36e"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.034126 5957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:27.036792 5972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.038210 5957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:27.038347 5957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "dc123d4493f844ceacf13590d301db8c"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8121d1604a29b9a6bcfff1ee7aa1656a"
server_key_iv: "d957616b75a7ad994860a51995bee36e"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.038450 5957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:27.049943 5957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:27.053452 5957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:27.053627 5957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:27.060817 5957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42721
I20260929 14:42:27.061144 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42721 every 8 connection(s)
I20260929 14:42:27.061996 5957 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:27.062817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 5957
I20260929 14:42:27.062952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:27.063232 392 external_mini_cluster.cc:1428] Setting key ab0bfb4a6003938c96d5dbc4508b4f40
I20260929 14:42:27.065562 6025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:27.072474 6025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c: Bootstrap starting.
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:27.075281 6025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:27.076093 6025 log.cc:824] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:27.078850 6025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c: No bootstrap required, opened a new log
I20260929 14:42:27.080227 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.064835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.0.98.62:42721)
0929 14:42:27.065356 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.065379 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.065418 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:42:27.066221 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.066242 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.066657 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.067013 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.067921 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.069196 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.070410 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.070946 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.071084 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.074224 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.074231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.074246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.074286 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.076344 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.076922 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.076930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.076937 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.077115 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.077515 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.077518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.077520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.078190 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.078396 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.078737 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.079086 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":144,"threads_started":1}
I20260929 14:42:27.082739 6025 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260929 14:42:27.083058 6025 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:27.083128 6025 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc123d4493f844ceacf13590d301db8c, State: Initialized, Role: FOLLOWER
I20260929 14:42:27.083562 6025 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260929 14:42:27.083709 6025 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:27.083776 6025 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:27.083874 6025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:27.084915 6025 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260929 14:42:27.085322 6025 leader_election.cc:302] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc123d4493f844ceacf13590d301db8c; no voters:
I20260929 14:42:27.085698 6025 leader_election.cc:288] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:27.085806 6030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:27.085971 6030 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [term 1 LEADER]: Becoming Leader. State: Replica: dc123d4493f844ceacf13590d301db8c, State: Running, Role: LEADER
I20260929 14:42:27.086277 6030 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260929 14:42:27.087100 6025 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:27.087577 6030 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc123d4493f844ceacf13590d301db8c. Latest consensus state: current_term: 1 leader_uuid: "dc123d4493f844ceacf13590d301db8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } } }
I20260929 14:42:27.087695 6030 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:27.087972 6031 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc123d4493f844ceacf13590d301db8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc123d4493f844ceacf13590d301db8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } } }
I20260929 14:42:27.088079 6031 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:27.088208 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:27.091166 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:27.096571 6034 catalog_manager.cc:1347] Generated new cluster ID: 7dfac42f8f274c70bef223db60b6e177
I20260929 14:42:27.096658 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:27.127126 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:27.128453 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:27.137364 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc123d4493f844ceacf13590d301db8c: Generated new TSK 0
I20260929 14:42:27.138238 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:27.176291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:27.290354 6051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:27.290649 6051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:27.290694 6051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:27.294951 6051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:27.295029 6051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:27.295099 6051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:27.300522 6051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:27.301671 6051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:27.302738 6051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:27.312665 6059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.312672 6056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.312829 6057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:27.312919 6051 server_base.cc:1034] running on GCE node
I20260929 14:42:27.313436 6051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:27.314129 6051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:27.315315 6051 hybrid_clock.cc:648] HybridClock initialized: now 1790692947315293 us; error 36 us; skew 500 ppm
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.323387 6051 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:27.324915 6051 webserver.cc:460] Webserver started at http://127.0.98.1:33785/ using document root <none> and password file <none>
I20260929 14:42:27.325726 6051 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:27.325830 6051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:27.326098 6051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:27.329144 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fd39294065a42d482892e9818de8127"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "f58d00efe8cf79e8b6af5a7f3edbdb1b"
server_key_iv: "a9d4c8fd1409adf567a0b90ec08e00e0"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.329836 6051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fd39294065a42d482892e9818de8127"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "f58d00efe8cf79e8b6af5a7f3edbdb1b"
server_key_iv: "a9d4c8fd1409adf567a0b90ec08e00e0"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.333839 6051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:27.336360 6066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.337683 6051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:27.337788 6051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "2fd39294065a42d482892e9818de8127"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "f58d00efe8cf79e8b6af5a7f3edbdb1b"
server_key_iv: "a9d4c8fd1409adf567a0b90ec08e00e0"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.337898 6051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:27.355427 6051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:27.358577 6051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:27.358744 6051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:27.359309 6051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:27.360144 6051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:27.360191 6051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.360234 6051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:27.360255 6051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.369109 6051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43533
I20260929 14:42:27.369132 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43533 every 8 connection(s)
I20260929 14:42:27.370041 6051 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:27.373149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6051
I20260929 14:42:27.373327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:27.373629 392 external_mini_cluster.cc:1428] Setting key dfa72ac5c2e553c29c85705514f1f131
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:27.385092 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.371978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46609 (local address 127.0.98.62:42721)
0929 14:42:27.372105 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.372108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.373095 (+ 987us) server_negotiation.cc:407] Connection header received
0929 14:42:27.374806 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.374809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.374854 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.374937 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.376535 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.377416 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.378678 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.378920 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.379009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.381516 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.381521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.381524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.381555 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.383005 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.383473 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.383476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.383477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.383550 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.383905 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.383908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.383910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.384097 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.384184 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.384748 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.384916 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:27.386119 6182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.372354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.1:46609)
0929 14:42:27.372896 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:27.372964 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:27.374583 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:27.375141 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:27.375150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:27.375657 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:27.376363 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.376383 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.377539 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.377542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.378536 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.378546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.379066 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.379070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.379271 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.379840 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:27.379856 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:27.381320 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:27.383112 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.383118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.383131 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.383393 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.383622 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.383624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.383626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.383801 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.384241 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:27.384247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:27.384498 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 14:42:27.384672 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:42:27.385129 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260929 14:42:27.387660 6180 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
I20260929 14:42:27.387935 6180 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:27.388628 6180 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260929 14:42:27.390372 5986 ts_manager.cc:194] Registered new tserver with Master: 2fd39294065a42d482892e9818de8127 (127.0.98.1:43533)
I20260929 14:42:27.391523 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:27.404748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:27.515637 6185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:27.515954 6185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:27.516000 6185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:27.520543 6185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:27.520628 6185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:27.520702 6185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:27.528189 6185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:27.529695 6185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:27.530869 6185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:27.540587 6193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.540704 6191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:27.540731 6185 server_base.cc:1034] running on GCE node
W20260929 14:42:27.540716 6190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:27.541360 6185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:27.542068 6185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:27.543234 6185 hybrid_clock.cc:648] HybridClock initialized: now 1790692947543213 us; error 39 us; skew 500 ppm
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.546293 6185 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:27.547320 6185 webserver.cc:460] Webserver started at http://127.0.98.2:41629/ using document root <none> and password file <none>
I20260929 14:42:27.547892 6185 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:27.547964 6185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:27.548156 6185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:27.550356 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "07ac057caea242a9b5c912b967647db3"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8e70b23096a18d7243e69a561ee9c231"
server_key_iv: "313ee3ebbff030b645d4dfffd962d350"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.550938 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "07ac057caea242a9b5c912b967647db3"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8e70b23096a18d7243e69a561ee9c231"
server_key_iv: "313ee3ebbff030b645d4dfffd962d350"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.555126 6185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:42:27.557843 6200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.559237 6185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:27.559362 6185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "07ac057caea242a9b5c912b967647db3"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "8e70b23096a18d7243e69a561ee9c231"
server_key_iv: "313ee3ebbff030b645d4dfffd962d350"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.559477 6185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:27.580032 6185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:27.583420 6185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:27.583618 6185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:27.584231 6185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:27.585145 6185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:27.585196 6185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.585250 6185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:27.585274 6185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.594715 6185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36869
I20260929 14:42:27.594743 6313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36869 every 8 connection(s)
I20260929 14:42:27.595687 6185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:27.601538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6185
I20260929 14:42:27.601675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:27.601985 392 external_mini_cluster.cc:1428] Setting key a45a981abc8ba75869ccb07c34c3e81b
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:27.611869 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.597580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54509 (local address 127.0.98.62:42721)
0929 14:42:27.597814 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.597818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.598739 (+ 921us) server_negotiation.cc:407] Connection header received
0929 14:42:27.600393 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.600397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.600450 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.600526 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.602364 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.603036 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.604984 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.605225 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.605339 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.608020 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.608025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.608027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.608058 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.609462 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.610076 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.610079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.610080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.610197 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.610617 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.610621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.610622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.610816 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.610905 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.611590 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.611735 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 14:42:27.612774 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.598033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.2:54509)
0929 14:42:27.598500 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:27.598622 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:27.600145 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:27.600849 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:27.600903 (+ 54us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:27.601376 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:27.602141 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.602166 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.603186 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.603190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.604783 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.604797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.605405 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.605410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.605616 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.606342 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:27.606360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:27.607817 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:27.609585 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.609592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.609611 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.609979 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.610284 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.610287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.610289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.610482 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.610927 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:27.610934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:27.611303 (+ 369us) client_negotiation.cc:769] Sending connection context
0929 14:42:27.611522 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 14:42:27.611904 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260929 14:42:27.614058 6314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:42:27.614327 6314 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:27.614913 6314 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260929 14:42:27.615993 5986 ts_manager.cc:194] Registered new tserver with Master: 07ac057caea242a9b5c912b967647db3 (127.0.98.2:36869)
I20260929 14:42:27.616648 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:27.632311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:27.745991 6319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:27.746279 6319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:27.746321 6319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:27.750578 6319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:27.750658 6319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:27.750720 6319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:27.756270 6319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:27.757507 6319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:27.758615 6319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:27.768702 6319 server_base.cc:1034] running on GCE node
W20260929 14:42:27.768718 6327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.768731 6324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:27.768903 6325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:27.769354 6319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:27.770013 6319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:27.771209 6319 hybrid_clock.cc:648] HybridClock initialized: now 1790692947771190 us; error 33 us; skew 500 ppm
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.774152 6319 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:27.775120 6319 webserver.cc:460] Webserver started at http://127.0.98.3:38963/ using document root <none> and password file <none>
I20260929 14:42:27.775671 6319 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:27.775738 6319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:27.775935 6319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:27.778015 6319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4e6a8413133433b8115acff688ea575"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "0580241325130417a16488c3a55718b6"
server_key_iv: "a89cb0f1a6e2806f58871196420b69a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.778594 6319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4e6a8413133433b8115acff688ea575"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "0580241325130417a16488c3a55718b6"
server_key_iv: "a89cb0f1a6e2806f58871196420b69a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.782555 6319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:27.785115 6334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.786435 6319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:27.786547 6319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "e4e6a8413133433b8115acff688ea575"
format_stamp: "Formatted at 2026-09-29 14:42:27 on dist-test-slave-g8hp"
server_key: "0580241325130417a16488c3a55718b6"
server_key_iv: "a89cb0f1a6e2806f58871196420b69a9"
server_key_version: "encryptionkey@0"
I20260929 14:42:27.786710 6319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:27.796514 6319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:27.799867 6319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:27.800057 6319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:27.800678 6319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:27.801594 6319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:27.801644 6319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.801699 6319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:27.801728 6319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:27.810617 6319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40913
I20260929 14:42:27.810642 6447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40913 every 8 connection(s)
I20260929 14:42:27.811501 6319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:27.818934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6319
I20260929 14:42:27.819042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:27.819324 392 external_mini_cluster.cc:1428] Setting key 2faa0e390f392e3d8b4ea2e98f7d329c
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:27.826629 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.813360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57641 (local address 127.0.98.62:42721)
0929 14:42:27.813518 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.813522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.814392 (+ 870us) server_negotiation.cc:407] Connection header received
0929 14:42:27.816170 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.816174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.816238 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.816330 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.817985 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.818553 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.820081 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.820420 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.820519 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.823034 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.823037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.823039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.823105 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.824421 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.824968 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.824971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.824972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.825102 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.825557 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.825561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.825562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.825753 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.825829 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.826391 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.826503 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 14:42:27.827404 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.813770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.3:57641)
0929 14:42:27.814229 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:27.814279 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:27.815853 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:27.816560 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:27.816581 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:27.817125 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:27.817778 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.817800 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.818686 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.818689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.819941 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:27.819953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.820575 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.820580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:27.820783 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.821429 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:27.821449 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:27.822880 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:27.824523 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.824528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.824541 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.824847 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.825276 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:27.825279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:27.825281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:27.825446 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:27.825863 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:27.825869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:27.826130 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 14:42:27.826297 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 14:42:27.826577 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260929 14:42:27.828588 6448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
I20260929 14:42:27.828837 6448 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:27.829411 6448 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260929 14:42:27.830423 5986 ts_manager.cc:194] Registered new tserver with Master: e4e6a8413133433b8115acff688ea575 (127.0.98.3:40913)
I20260929 14:42:27.830960 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57641
I20260929 14:42:27.833281 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/test-admin.keytab.
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.850237 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 14:42:27.853078 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:27.851753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.98.1:43533)
0929 14:42:27.852041 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.852057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.852081 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:42:27.852130 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.852136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.852324 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.852468 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.852903 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58812: BlockingRecv error: recv got EOF from 127.0.0.1:58812 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":91,"threads_started":1}
W20260929 14:42:27.853214 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.854328 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.855407 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:27.866168 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.856468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.0.98.1:43533)
0929 14:42:27.856593 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.856596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.856682 (+ 86us) server_negotiation.cc:407] Connection header received
0929 14:42:27.856783 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.856785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.856825 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.856928 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.857442 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.858249 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.859212 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.859689 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.859785 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.862189 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.862200 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.862214 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.862240 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.864152 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.864665 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.864671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.864672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.864762 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.865165 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.865168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.865169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.865445 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.865613 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.865834 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.865987 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 14:42:27.870345 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:27.869271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.0.98.1:43533)
0929 14:42:27.869412 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.869416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.869516 (+ 100us) server_negotiation.cc:407] Connection header received
0929 14:42:27.869662 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.869665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.869709 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.869771 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.870218 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58836: BlockingRecv error: recv got EOF from 127.0.0.1:58836 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 14:42:27.870500 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.871577 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.872502 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:27.881722 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.873437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.0.98.1:43533)
0929 14:42:27.873574 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.873578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.873685 (+ 107us) server_negotiation.cc:407] Connection header received
0929 14:42:27.873832 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.873836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.873878 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.873949 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.874457 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.874979 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.875915 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.876197 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.876291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.878646 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.878649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.878651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.878682 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.880087 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.880517 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.880520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.880522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.880620 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.880897 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.880901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.880903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.881076 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.881242 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.881459 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.881570 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 14:42:27.885013 6455 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:27.883946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.0.98.1:43533)
0929 14:42:27.884084 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.884087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.884202 (+ 115us) server_negotiation.cc:407] Connection header received
0929 14:42:27.884321 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.884324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.884362 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.884423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.884848 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58846: BlockingRecv error: recv got EOF from 127.0.0.1:58846 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 14:42:27.885244 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790692915364895-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.886228 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:27.887050 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[5944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:42:27.895799 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:27.888254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.0.98.1:43533)
0929 14:42:27.888358 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:27.888361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:27.888500 (+ 139us) server_negotiation.cc:407] Connection header received
0929 14:42:27.888603 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:27.888606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:27.888648 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:27.888705 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:27.889320 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.889810 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.890658 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:27.890870 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:27.890961 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:27.893085 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:27.893088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:27.893090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:27.893115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:27.894406 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.894761 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.894764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.894765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.894829 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:27.895105 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:27.895108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:27.895109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:27.895269 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:27.895362 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:27.895566 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 14:42:27.895656 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 14:42:27.896888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6051
I20260929 14:42:27.901965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6185
I20260929 14:42:27.907444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6319
I20260929 14:42:27.912379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 5957
2026-09-29T14:42:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2104 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[6468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[6468](info): set up 2 sockets
Sep 29 14:42:27 dist-test-slave-g8hp krb5kdc[6468](info): commencing operation
krb5kdc: starting...
W20260929 14:42:28.869134 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:28 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:42:29.000974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:29.110971 6481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:29.111243 6481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:29.111282 6481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:29.115383 6481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:29.115468 6481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:29.115486 6481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:29.115501 6481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:29.115515 6481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:29.120920 6481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:29.121984 6481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:29.122964 6481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:29.131408 6489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.131490 6486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.131635 6487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.131639 6481 server_base.cc:1034] running on GCE node
I20260929 14:42:29.132305 6481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:29.133523 6481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:29.134709 6481 hybrid_clock.cc:648] HybridClock initialized: now 1790692949134681 us; error 40 us; skew 500 ppm
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:29.137686 6481 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:42:29.138630 6481 webserver.cc:460] Webserver started at http://127.0.98.62:43435/ using document root <none> and password file <none>
I20260929 14:42:29.139194 6481 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:29.139261 6481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:29.139477 6481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:29.141583 6481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "e506138610264abd9f8ce3034ffddafc"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "487fee0d81cb7857c93b5465c21a6024"
server_key_iv: "1f098df0d08097d1a0f80d09b36d4e33"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.142127 6481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "e506138610264abd9f8ce3034ffddafc"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "487fee0d81cb7857c93b5465c21a6024"
server_key_iv: "1f098df0d08097d1a0f80d09b36d4e33"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.145798 6481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:29.148569 6496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.150069 6481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:29.150197 6481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "e506138610264abd9f8ce3034ffddafc"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "487fee0d81cb7857c93b5465c21a6024"
server_key_iv: "1f098df0d08097d1a0f80d09b36d4e33"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.150301 6481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:29.178712 6481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:29.182333 6481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:29.182545 6481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:29.190014 6481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40905
I20260929 14:42:29.190274 6548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40905 every 8 connection(s)
I20260929 14:42:29.191056 6481 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:29.194092 6549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:29.197831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6481
I20260929 14:42:29.197981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:29.198230 392 external_mini_cluster.cc:1428] Setting key 6255c427abe1527de3117e4fe8304a0e
I20260929 14:42:29.200215 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Bootstrap starting.
I20260929 14:42:29.202824 6549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:29.203569 6549 log.cc:824] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:29.205891 6549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: No bootstrap required, opened a new log
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:29.209367 6549 raft_consensus.cc:348] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } }
I20260929 14:42:29.209627 6549 raft_consensus.cc:374] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:29.209686 6549 raft_consensus.cc:729] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e506138610264abd9f8ce3034ffddafc, State: Initialized, Role: FOLLOWER
I20260929 14:42:29.210202 6549 consensus_queue.cc:260] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } }
I20260929 14:42:29.210359 6549 raft_consensus.cc:388] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:29.210414 6549 raft_consensus.cc:482] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:29.210518 6549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:29.211585 6549 raft_consensus.cc:504] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } }
I20260929 14:42:29.211911 6549 leader_election.cc:302] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e506138610264abd9f8ce3034ffddafc; no voters:
I20260929 14:42:29.212209 6549 leader_election.cc:288] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:29.212292 6554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:29.212446 6554 raft_consensus.cc:686] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 LEADER]: Becoming Leader. State: Replica: e506138610264abd9f8ce3034ffddafc, State: Running, Role: LEADER
I20260929 14:42:29.212744 6554 consensus_queue.cc:237] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } }
I20260929 14:42:29.213645 6549 sys_catalog.cc:564] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:29.214007 6555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e506138610264abd9f8ce3034ffddafc. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } }
I20260929 14:42:29.214120 6555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:29.214313 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } }
I20260929 14:42:29.214402 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:29.214540 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.199711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.98.62:40905)
0929 14:42:29.200145 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:29.200161 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:29.200197 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:29.200924 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:29.200943 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:29.201306 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:29.201634 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:29.202539 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.203678 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.204682 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.205267 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.205500 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.208213 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:29.208226 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:29.208241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:29.208271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:29.210829 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.211290 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.211296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.211302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.211455 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.211763 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.211767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.211769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.212215 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:29.212413 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:29.212923 (+ 510us) server_negotiation.cc:299] Negotiation successful
0929 14:42:29.213295 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
W20260929 14:42:29.218772 6569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:29.218868 6569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:29.218950 6558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:29.219825 6558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:29.226131 6558 catalog_manager.cc:1347] Generated new cluster ID: 62c6a16bba614ba7a14d79c04bba8b0f
I20260929 14:42:29.226218 6558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:29.242450 6558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:29.243402 6558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:29.253340 6558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Generated new TSK 0
I20260929 14:42:29.254076 6558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:42:29.288887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:29.414638 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:29.414989 6575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:29.415079 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:29.419301 6575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:29.419379 6575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:29.419445 6575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:29.424800 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:29.425911 6575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:29.426956 6575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:29.436847 6575 server_base.cc:1034] running on GCE node
W20260929 14:42:29.436816 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.436988 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.437198 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.437611 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:29.438237 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:29.439385 6575 hybrid_clock.cc:648] HybridClock initialized: now 1790692949439359 us; error 40 us; skew 500 ppm
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:29.442173 6575 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:42:29.443025 6575 webserver.cc:460] Webserver started at http://127.0.98.1:40351/ using document root <none> and password file <none>
I20260929 14:42:29.443535 6575 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:29.443622 6575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:29.443790 6575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:29.445848 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "91aae8b7928041d9876cfc190512f8e8"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "535e377249ec059c0356671e2969babd"
server_key_iv: "37c9444320ce947650344912dd57f191"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.446360 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91aae8b7928041d9876cfc190512f8e8"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "535e377249ec059c0356671e2969babd"
server_key_iv: "37c9444320ce947650344912dd57f191"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.450098 6575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 14:42:29.452608 6590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.453925 6575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:29.454023 6575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "91aae8b7928041d9876cfc190512f8e8"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "535e377249ec059c0356671e2969babd"
server_key_iv: "37c9444320ce947650344912dd57f191"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.454133 6575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:29.476179 6575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:29.480450 6575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:29.480626 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:29.481259 6575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:29.482084 6575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:29.482138 6575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.482177 6575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:29.482191 6575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.490608 6575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44371
I20260929 14:42:29.490645 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44371 every 8 connection(s)
I20260929 14:42:29.491542 6575 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:29.496707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6575
I20260929 14:42:29.496838 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:29.497150 392 external_mini_cluster.cc:1428] Setting key 79741d5863c62fb6297c4d3403439097
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:29.507243 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.493484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49485 (local address 127.0.98.62:40905)
0929 14:42:29.493622 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:29.493627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:29.494501 (+ 874us) server_negotiation.cc:407] Connection header received
0929 14:42:29.495897 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:29.495900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:29.495951 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:29.496025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:29.497661 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.498406 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.500082 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.500333 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.500448 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.503224 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:29.503228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:29.503231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:29.503261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:29.505062 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.505642 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.505646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.505648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.505739 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.506053 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.506057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.506058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.506297 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:29.506404 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:29.506984 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 14:42:29.507112 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:29.508193 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.493875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:49485)
0929 14:42:29.494340 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:29.494383 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:29.495678 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:29.496187 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:29.496194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:29.496731 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:29.497472 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.497494 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.498552 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.498555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.499943 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.499955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.500508 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.500513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.500712 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.501380 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:29.501397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:29.503029 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:29.505183 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.505188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.505204 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.505540 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.505827 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.505829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.505831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.505968 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.506445 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:29.506450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:29.506717 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 14:42:29.506927 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 14:42:29.507222 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260929 14:42:29.509680 6704 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:29.509964 6704 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:29.510478 6704 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:29.512041 6513 ts_manager.cc:194] Registered new tserver with Master: 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371)
I20260929 14:42:29.513587 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:42:29.529775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:29.639838 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:29.640122 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:29.640168 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:29.644332 6709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:29.644411 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:29.644485 6709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:29.649982 6709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:29.651194 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:29.652279 6709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:29.662329 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.662328 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.662456 6709 server_base.cc:1034] running on GCE node
W20260929 14:42:29.662328 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.663096 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:29.663748 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:29.664925 6709 hybrid_clock.cc:648] HybridClock initialized: now 1790692949664898 us; error 34 us; skew 500 ppm
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:29.667755 6709 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:42:29.668675 6709 webserver.cc:460] Webserver started at http://127.0.98.2:44931/ using document root <none> and password file <none>
I20260929 14:42:29.669267 6709 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:29.669335 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:29.669523 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:29.671586 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "0828b6e790f746eca979c0d54b186fed"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "66fa18cefe5f6cad1b8759aac6ba5c8f"
server_key_iv: "ada398be001761be8123db6561c78bc0"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.672135 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0828b6e790f746eca979c0d54b186fed"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "66fa18cefe5f6cad1b8759aac6ba5c8f"
server_key_iv: "ada398be001761be8123db6561c78bc0"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.676311 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:29.679252 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.681043 6709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 14:42:29.681173 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "0828b6e790f746eca979c0d54b186fed"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "66fa18cefe5f6cad1b8759aac6ba5c8f"
server_key_iv: "ada398be001761be8123db6561c78bc0"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.681293 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:29.702941 6709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:29.706281 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:29.706480 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:29.707077 6709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:29.707918 6709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:29.707966 6709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.708017 6709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:29.708045 6709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.717411 6709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39919
I20260929 14:42:29.717443 6837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39919 every 8 connection(s)
I20260929 14:42:29.718290 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:29.727236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6709
I20260929 14:42:29.727339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:29.727591 392 external_mini_cluster.cc:1428] Setting key 4cd032e4d475468731ad7380ec9076a5
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:29.732192 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.720021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53595 (local address 127.0.98.62:40905)
0929 14:42:29.720179 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:29.720184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:29.721132 (+ 948us) server_negotiation.cc:407] Connection header received
0929 14:42:29.722488 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:29.722491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:29.722540 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:29.722615 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:29.723829 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.724411 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.725658 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.725904 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.725984 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.728580 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:29.728583 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:29.728585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:29.728609 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:29.730155 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.730743 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.730746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.730747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.730825 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.731145 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.731149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.731151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.731326 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:29.731409 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:29.731911 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 14:42:29.732034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:42:29.733134 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.720451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:53595)
0929 14:42:29.720981 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:29.721028 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:29.722268 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:29.722794 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:29.722801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:29.723184 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:29.723683 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.723697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.724536 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.724539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.725540 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.725549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.726062 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.726065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.726215 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.726738 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:29.726754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:29.728438 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:29.730275 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.730280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.730293 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.730628 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.730909 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.730912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.730915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.731059 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.731427 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:29.731434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:29.731678 (+ 244us) client_negotiation.cc:769] Sending connection context
0929 14:42:29.731847 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 14:42:29.732201 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":124,"threads_started":1}
I20260929 14:42:29.734617 6838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
I20260929 14:42:29.734879 6838 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:29.735380 6838 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:29.736460 6513 ts_manager.cc:194] Registered new tserver with Master: 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:29.737196 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:42:29.757256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:29.867374 6843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:29.867647 6843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:29.867690 6843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:29.871774 6843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:29.871863 6843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:29.871925 6843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:29.877362 6843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:29.878446 6843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:29.879446 6843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:29.894189 6849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:29.894186 6851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.894444 6843 server_base.cc:1034] running on GCE node
W20260929 14:42:29.894189 6848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:29.894953 6843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:29.895607 6843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:29.896782 6843 hybrid_clock.cc:648] HybridClock initialized: now 1790692949896760 us; error 30 us; skew 500 ppm
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:29.899842 6843 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:42:29.900854 6843 webserver.cc:460] Webserver started at http://127.0.98.3:42861/ using document root <none> and password file <none>
I20260929 14:42:29.901437 6843 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:29.901502 6843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:29.901691 6843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:29.903767 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "0cbe9e1104fb4153ad340681d9da5760"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "8756ce17eab5f372a607441022d6aaa8"
server_key_iv: "41ead26aae190f2f0d93c901be75a318"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.904320 6843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbe9e1104fb4153ad340681d9da5760"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "8756ce17eab5f372a607441022d6aaa8"
server_key_iv: "41ead26aae190f2f0d93c901be75a318"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.907982 6843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260929 14:42:29.910531 6858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.911952 6843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:29.912071 6843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0cbe9e1104fb4153ad340681d9da5760"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "8756ce17eab5f372a607441022d6aaa8"
server_key_iv: "41ead26aae190f2f0d93c901be75a318"
server_key_version: "encryptionkey@0"
I20260929 14:42:29.912173 6843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:29.927086 6843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:29.930135 6843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:29.930303 6843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:29.930884 6843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:29.931690 6843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:29.931736 6843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.931787 6843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:29.931814 6843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:29.940397 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33323 every 8 connection(s)
I20260929 14:42:29.940470 6843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33323
I20260929 14:42:29.941457 6843 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:29.943354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6843
I20260929 14:42:29.943460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:29.943720 392 external_mini_cluster.cc:1428] Setting key ad7ce43dc09fd9588c2d6e3a08fc8082
Sep 29 14:42:29 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:29.954828 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.943237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49547 (local address 127.0.98.62:40905)
0929 14:42:29.943407 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:29.943412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:29.944249 (+ 837us) server_negotiation.cc:407] Connection header received
0929 14:42:29.945618 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:29.945622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:29.945672 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:29.945736 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:29.946835 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.947367 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.948497 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.948709 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.948786 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.951304 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:29.951307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:29.951309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:29.951335 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:29.952674 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.953233 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.953236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.953237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.953316 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:29.953684 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:29.953687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:29.953688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:29.953895 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:29.953969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:29.954577 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 14:42:29.954701 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 14:42:29.955778 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:29.943638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:49547)
0929 14:42:29.944104 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:29.944154 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:29.945434 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:29.945901 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:29.945908 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:29.946248 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:29.946710 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.946723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.947471 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.947473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.948394 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:29.948402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:29.948900 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:29.948905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:29.949064 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:29.949655 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:29.949672 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:29.951137 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:29.952791 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.952796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.952812 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.953151 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.953437 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:29.953441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:29.953443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:29.953605 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:29.953994 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:29.954002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:29.954275 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 14:42:29.954536 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 14:42:29.954860 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
I20260929 14:42:29.957012 6972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
I20260929 14:42:29.957270 6972 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:29.957859 6972 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:29.958933 6512 ts_manager.cc:194] Registered new tserver with Master: 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
I20260929 14:42:29.959509 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49547
I20260929 14:42:29.969741 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 14:42:30.000618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39265
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40905,127.0.98.61:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:30.114512 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:30.114789 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:30.114830 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:30.119043 6977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:30.119133 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:30.119156 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:30.119180 6977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:30.119199 6977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:30.126736 6977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40905,127.0.98.61:39265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
W20260929 14:42:30.126950 6977 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40905,127.0.98.61:39265'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 14:42:30.127799 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:30.128777 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.137017 6982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.137069 6977 server_base.cc:1034] running on GCE node
W20260929 14:42:30.137017 6985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.137017 6983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.137869 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.138971 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.140153 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790692950140073 us; error 95 us; skew 500 ppm
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.142946 6977 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:42:30.143879 6977 webserver.cc:460] Webserver started at http://127.0.98.61:40729/ using document root <none> and password file <none>
I20260929 14:42:30.144407 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.144470 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.144665 6977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:30.146723 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/instance:
uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "b6d172bac0eba7e6c495e16b890a87ac"
server_key_iv: "5f3c5a79a78d187aa0d65a0537fde967"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.147271 6977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal/instance:
uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "b6d172bac0eba7e6c495e16b890a87ac"
server_key_iv: "5f3c5a79a78d187aa0d65a0537fde967"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.151022 6977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:30.153801 6992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.155328 6977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:30.155457 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "b6d172bac0eba7e6c495e16b890a87ac"
server_key_iv: "5f3c5a79a78d187aa0d65a0537fde967"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.155562 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.172935 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.176393 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:30.176570 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.183642 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39265
I20260929 14:42:30.183872 7044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39265 every 8 connection(s)
I20260929 14:42:30.184654 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:42:30.187069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 6977
I20260929 14:42:30.187810 7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:30.192296 7045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:30.203101 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.188671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.0.98.61:39265)
0929 14:42:30.189089 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.189105 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.189140 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:42:30.189844 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.189867 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.190310 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.190687 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.191496 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.193101 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.194157 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.194688 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.194833 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.197738 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.197745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.197757 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.197789 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.199792 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.200303 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.200309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.200316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.200449 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.200735 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.200738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.200740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.201263 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.201445 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.201776 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.202071 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":148,"threads_started":1}
I20260929 14:42:30.204222 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.194122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44806)
0929 14:42:30.195000 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.195048 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.195218 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.195553 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.195558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.195999 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.196149 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.196156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.197009 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.197012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.197743 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.197751 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.198309 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.198313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.198407 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.198890 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.198904 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.200614 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.202549 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.202556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.202560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.202753 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.203025 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.203028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.203030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.203105 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.203647 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.203652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.203798 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.203920 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.204061 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":112,"threads_started":1}
I20260929 14:42:30.204273 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.194195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.62:40905)
0929 14:42:30.194314 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.194318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.195117 (+ 799us) server_negotiation.cc:407] Connection header received
0929 14:42:30.195373 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.195376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.195426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.195492 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.196282 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.196892 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.197860 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.198183 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.198305 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.200786 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.200791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.200793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.200822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.202444 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.202857 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.202861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.202863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.202942 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.203282 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.203286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.203287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.203545 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.203641 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.203992 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.204108 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:30.205793 7045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:42:30.214342 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.206233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33952)
0929 14:42:30.206386 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.206401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.206480 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.206713 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.206716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.206885 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.206978 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.206983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.208039 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.208042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.208582 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.208591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.209082 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.209085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.209163 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.209664 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.209676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.210949 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.212643 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.212645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.212647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.212853 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.213160 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.213163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.213164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.213323 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.213882 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.213885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.213998 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.214099 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.214207 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 14:42:30.214509 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.206428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.98.61:39265)
0929 14:42:30.206532 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.206534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.206545 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:30.206570 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.206572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.206602 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.206661 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.207070 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.207929 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.208681 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.208962 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.209044 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.211094 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.211098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.211100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.211120 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.212541 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.212966 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.212969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.212971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.213040 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.213478 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.213482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.213484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.213663 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.213752 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.214176 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.214280 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:30.217576 7045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Bootstrap starting.
I20260929 14:42:30.219861 7045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:30.220572 7045 log.cc:824] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:30.222582 7045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: No bootstrap required, opened a new log
I20260929 14:42:30.225644 7045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } }
I20260929 14:42:30.225860 7045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:30.225895 7045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ea92adb9fa4235b9277ad0ee5d961b, State: Initialized, Role: FOLLOWER
I20260929 14:42:30.226361 7045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } }
I20260929 14:42:30.226853 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } } }
I20260929 14:42:30.227011 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:30.227571 7045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:42:30.232349 7062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:30.232456 7062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:30.234643 6977 master_runner.cc:428] Detected that this master 43ea92adb9fa4235b9277ad0ee5d961b is joining an existing cluster
I20260929 14:42:30.234704 6977 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39265
I20260929 14:42:30.244570 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.237378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33956)
0929 14:42:30.237629 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.237640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.237730 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.237975 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.237978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.238231 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.238356 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.238362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.239326 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.239329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.239981 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.239989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.240519 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.240523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.240609 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.241137 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.241150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.241546 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.243122 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.243127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.243301 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.243534 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.243539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.243604 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.244035 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.244038 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.244124 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.244231 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.244377 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":31,"thread_start_us":85,"threads_started":1}
I20260929 14:42:30.244596 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.237197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.98.61:39265)
0929 14:42:30.237306 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.237309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.237706 (+ 397us) server_negotiation.cc:407] Connection header received
0929 14:42:30.237799 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.237802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.237837 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.237895 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.238468 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.239208 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.240087 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.240407 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.240504 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.241657 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.241660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.241663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.241682 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.243030 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243378 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.243382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.243384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.243450 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243695 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.243699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.243700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.243920 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.244013 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.244274 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.244444 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":52}
I20260929 14:42:30.245002 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.236896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.0.98.62:40905)
0929 14:42:30.237014 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.237018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.237106 (+ 88us) server_negotiation.cc:407] Connection header received
0929 14:42:30.237230 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.237233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.237285 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.237371 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.237887 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.238599 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.239408 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.239744 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.239858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.241628 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.241633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.241636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.241663 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.243404 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243776 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.243780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.243782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.243858 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.244112 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.244116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.244117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.244314 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.244381 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.244706 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.244833 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:30.245395 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.236744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44816)
0929 14:42:30.237018 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.237032 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.237135 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.237400 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.237403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.237629 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.237766 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.237774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.238687 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.238690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.239293 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.239303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.239860 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.239864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.239990 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.240983 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.240999 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.241490 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.243494 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.243499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.243698 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.243934 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.243937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.243939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.244016 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.244445 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.244448 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.244537 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.245122 (+ 585us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.245244 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":25,"thread_start_us":114,"threads_started":1}
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:42:30.259680 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.249771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33970)
0929 14:42:30.250125 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.250177 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.250317 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.250625 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.250630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.251029 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.251137 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.251143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.252127 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.252129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.252835 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.252846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.253423 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.253427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.253509 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.254143 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.254155 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.255673 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.257531 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.257541 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.257549 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.257866 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.258143 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.258145 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.258147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.258239 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.258974 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.258982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.259120 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.259268 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.259489 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260929 14:42:30.260339 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.249850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.0.98.61:39265)
0929 14:42:30.249963 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.249967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.250249 (+ 282us) server_negotiation.cc:407] Connection header received
0929 14:42:30.250453 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.250456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.250507 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.250586 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.251239 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.252007 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.252990 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.253323 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.253421 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.255802 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.255806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.255809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.255843 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.257419 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.257962 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.257966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.257968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.258052 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.258317 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.258320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.258322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.258810 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.258916 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.260072 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.260192 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:42:30.262252 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39265: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39265
}
attrs {
promote: true
}
}
dest_uuid: "e506138610264abd9f8ce3034ffddafc"
cas_config_opid_index: -1
I20260929 14:42:30.262951 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 14:42:30.265017 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } }
I20260929 14:42:30.265745 7019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:30.267655 7074 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39265
I20260929 14:42:30.267668 7019 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Refusing update from remote peer e506138610264abd9f8ce3034ffddafc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 14:42:30.267794 7051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:30.268004 7051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260929 14:42:30.269253 7051 leader_election.cc:288] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905)
I20260929 14:42:30.269573 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:30.269773 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e506138610264abd9f8ce3034ffddafc" is_pre_election: true
I20260929 14:42:30.270416 6977 master.cc:537] Master@127.0.98.61:39265 shutting down...
I20260929 14:42:30.270646 6993 leader_election.cc:302] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 43ea92adb9fa4235b9277ad0ee5d961b; no voters: e506138610264abd9f8ce3034ffddafc
I20260929 14:42:30.271373 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } }
I20260929 14:42:30.271487 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:30.271569 6497 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 14:42:30.272339 7051 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:42:30.273530 6497 consensus_peers.cc:487] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc -> Peer 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61:39265): Couldn't send request to peer 43ea92adb9fa4235b9277ad0ee5d961b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:42:30.275264 6977 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:30.275549 6977 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:30.275648 6977 tablet_replica.cc:330] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: stopping tablet replica
I20260929 14:42:30.281313 6977 master.cc:552] Master@127.0.98.61:39265 shutdown complete.
I20260929 14:42:30.281368 6977 master_runner.cc:305] Clearing existing system tablet
I20260929 14:42:30.281591 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.281644 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.282908 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.284463 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.284726 6977 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.284840 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.284940 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "b6d172bac0eba7e6c495e16b890a87ac"
server_key_iv: "5f3c5a79a78d187aa0d65a0537fde967"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.287052 6977 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 14:42:30.288499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44347
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:42:30.289295 6977 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:42:30.289391 6977 log.cc:1196] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 14:42:30.289613 6977 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Deleting consensus metadata
I20260929 14:42:30.289817 6977 master_runner.cc:315] Copying system tablet from 127.0.98.62:40905
I20260929 14:42:30.290872 6977 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40905
I20260929 14:42:30.298245 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.291480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.62:40905)
0929 14:42:30.291602 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.291606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.291812 (+ 206us) server_negotiation.cc:407] Connection header received
0929 14:42:30.291946 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.291949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.291998 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.292062 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.292498 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.293090 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.293939 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.294160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.294250 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.295305 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.295308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.295311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.295338 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.296662 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.297122 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.297126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.297128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.297205 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.297457 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.297460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.297462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.297641 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.297725 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.297983 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.298089 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:30.298277 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.291381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44826)
0929 14:42:30.291712 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.291723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.291851 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.292091 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.292094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.292287 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.292380 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.292387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.293195 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.293199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.293813 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.293829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.294264 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.294267 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.294356 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.294841 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.294853 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.295165 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.296767 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.296770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.296773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.297034 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.297286 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.297289 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.297292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.297379 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.297728 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.297731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.297832 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.297916 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.298064 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":82,"threads_started":1}
I20260929 14:42:30.299041 6533 tablet_copy_service.cc:140] P e506138610264abd9f8ce3034ffddafc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 43ea92adb9fa4235b9277ad0ee5d961b ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44826)
I20260929 14:42:30.299173 6533 tablet_copy_service.cc:161] P e506138610264abd9f8ce3034ffddafc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 43ea92adb9fa4235b9277ad0ee5d961b at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44826: session id = 43ea92adb9fa4235b9277ad0ee5d961b-00000000000000000000000000000000
I20260929 14:42:30.300521 6533 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:42:30.302062 6977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:42:30.303965 6977 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: tablet copy: Starting download of 0 data blocks...
I20260929 14:42:30.304142 6977 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: tablet copy: Starting download of 1 WAL segments...
I20260929 14:42:30.305733 6977 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:42:30.307355 6533 tablet_copy_service.cc:342] P e506138610264abd9f8ce3034ffddafc: Request end of tablet copy session 43ea92adb9fa4235b9277ad0ee5d961b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44826
I20260929 14:42:30.307452 6533 tablet_copy_service.cc:433] P e506138610264abd9f8ce3034ffddafc: ending tablet copy session 43ea92adb9fa4235b9277ad0ee5d961b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 43ea92adb9fa4235b9277ad0ee5d961b
I20260929 14:42:30.314644 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.316370 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.317883 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.318192 6977 server_base.cc:1034] running on GCE node
W20260929 14:42:30.322126 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.322425 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.322736 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.323877 6977 hybrid_clock.cc:648] HybridClock initialized: now 1790692950323857 us; error 38 us; skew 500 ppm
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.325420 6977 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 14:42:30.325872 6977 webserver.cc:460] Webserver started at http://127.0.98.61:39185/ using document root <none> and password file <none>
I20260929 14:42:30.326052 6977 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.326089 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.327235 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.328080 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.328315 6977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.328387 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "43ea92adb9fa4235b9277ad0ee5d961b"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "b6d172bac0eba7e6c495e16b890a87ac"
server_key_iv: "5f3c5a79a78d187aa0d65a0537fde967"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.328464 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.336930 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.338653 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.343243 7150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39265 every 8 connection(s)
I20260929 14:42:30.343298 6977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39265
I20260929 14:42:30.343943 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:42:30.346030 7151 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:42:30.346905 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Bootstrap starting.
I20260929 14:42:30.355405 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:42:30.355687 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Bootstrap complete.
I20260929 14:42:30.356189 7151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } }
I20260929 14:42:30.356308 7151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43ea92adb9fa4235b9277ad0ee5d961b, State: Initialized, Role: LEARNER
I20260929 14:42:30.356417 7151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } }
I20260929 14:42:30.356736 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } }
I20260929 14:42:30.356887 7151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:30.356950 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: LEARNER
I20260929 14:42:30.360193 7165 catalog_manager.cc:1259] Loaded cluster ID: 62c6a16bba614ba7a14d79c04bba8b0f
I20260929 14:42:30.360247 7165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: loading cluster ID for follower catalog manager: success
I20260929 14:42:30.361860 7165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: acquiring CA information for follower catalog manager: success
I20260929 14:42:30.362962 7165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 14:42:30.432075 7081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:30.432374 7081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:30.432418 7081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:30.437008 7081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:30.437078 7081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:30.437098 7081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:30.437114 7081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:30.437143 7081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:30.442597 7081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:30.443671 7081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:30.444769 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.453238 7171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.453246 7170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.453337 7081 server_base.cc:1034] running on GCE node
W20260929 14:42:30.453238 7173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.454056 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.455250 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.456420 7081 hybrid_clock.cc:648] HybridClock initialized: now 1790692950456394 us; error 37 us; skew 500 ppm
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.459553 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:42:30.460386 7081 webserver.cc:460] Webserver started at http://127.0.98.60:35241/ using document root <none> and password file <none>
I20260929 14:42:30.460938 7081 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.461025 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.461203 7081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:30.463227 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/instance:
uuid: "9c3bdafc02074b0f949e37e84091d49f"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "854cc38423372c5b863b648cea9a4efa"
server_key_iv: "92f1e3948ade88a808b2748fbdedde0c"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.463743 7081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal/instance:
uuid: "9c3bdafc02074b0f949e37e84091d49f"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "854cc38423372c5b863b648cea9a4efa"
server_key_iv: "92f1e3948ade88a808b2748fbdedde0c"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.467463 7081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:30.469924 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.471236 7081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:30.471413 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "9c3bdafc02074b0f949e37e84091d49f"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "854cc38423372c5b863b648cea9a4efa"
server_key_iv: "92f1e3948ade88a808b2748fbdedde0c"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.471520 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.503082 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.506695 7081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:30.506877 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.514246 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44347
I20260929 14:42:30.514554 7232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44347 every 8 connection(s)
I20260929 14:42:30.515293 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:42:30.516145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7081
I20260929 14:42:30.516839 6704 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
I20260929 14:42:30.518784 7233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:30.522699 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:30.531822 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.517901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.98.60:44347)
0929 14:42:30.518415 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.518435 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.518478 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:42:30.519451 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.519485 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.519874 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.520230 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.521086 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.522256 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.523314 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.523790 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.523948 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.526793 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.526799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.526809 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.526836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.528619 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.529110 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.529117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.529123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.529232 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.529549 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.529553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.529555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.530114 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.530325 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.530604 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.530898 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":156,"threads_started":1}
I20260929 14:42:30.533046 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.524507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.0.98.62:40905)
0929 14:42:30.524635 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.524640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.524899 (+ 259us) server_negotiation.cc:407] Connection header received
0929 14:42:30.525029 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.525033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.525089 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.525181 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.525860 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.526440 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.527347 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.527570 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.527659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.529896 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.529901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.529904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.529933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.531264 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.531720 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.531723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.531724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.531834 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.532143 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.532146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.532148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.532342 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.532428 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.532741 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.532889 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:42:30.533071 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.524358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44832)
0929 14:42:30.524738 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.524788 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.524935 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.525235 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.525239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.525577 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.525729 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.525738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.526564 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.526567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.527218 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.527237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.527667 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.527670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.527782 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.528197 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.528207 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.529688 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.531372 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.531379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.531384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.531633 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.531958 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.531962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.531964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.532059 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.532442 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.532450 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.532558 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.532675 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.532892 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":142,"threads_started":1}
I20260929 14:42:30.534111 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 14:42:30.542055 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.534544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33974)
0929 14:42:30.534693 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.534704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.534759 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.535118 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.535121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.535269 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.535359 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.535364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.536229 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.536233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.536785 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.536793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.537232 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.537235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.537324 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.537754 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.537766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.538986 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.540671 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.540674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.540676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.540892 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.541132 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.541135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.541137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.541212 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.541567 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.541570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.541658 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.541740 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.541884 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 14:42:30.542057 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.534645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.61:39265)
0929 14:42:30.534933 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.534936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.534947 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:30.534980 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.534982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.535018 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.535090 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.535465 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.536132 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.536912 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.537137 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.537223 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.539122 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.539125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.539127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.539148 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.540571 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.540976 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.540980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.540981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.541053 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.541298 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.541302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.541303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.541481 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.541564 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.541801 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.541912 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260929 14:42:30.542979 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:42:30.551041 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.543444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36486)
0929 14:42:30.543553 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.543566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.543625 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.543864 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.543866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.543989 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.544055 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.544059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.545199 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.545201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.545708 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.545715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.546132 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.546134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.546194 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.546631 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.546642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.547890 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.549628 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.549631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.549632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.549818 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.550074 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.550077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.550078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.550161 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.550621 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.550624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.550705 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.550813 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.550915 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:42:30.551178 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.543534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.98.60:44347)
0929 14:42:30.543631 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.543634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.543668 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:42:30.543728 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.543730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.543764 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.543820 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.544166 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.545089 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.545818 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.546042 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.546115 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.548034 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.548037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.548039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.548060 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.549518 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.549900 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.549903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.549905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.549974 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.550330 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.550333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.550334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.550521 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.550603 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.550913 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.551011 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 14:42:30.553939 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Bootstrap starting.
I20260929 14:42:30.556140 7233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:30.556819 7233 log.cc:824] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:30.559077 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: No bootstrap required, opened a new log
I20260929 14:42:30.562258 7233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } }
I20260929 14:42:30.562480 7233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:30.562515 7233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3bdafc02074b0f949e37e84091d49f, State: Initialized, Role: FOLLOWER
I20260929 14:42:30.562901 7233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } }
I20260929 14:42:30.563334 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } } }
I20260929 14:42:30.563454 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:30.564057 7233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 14:42:30.568747 7250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:30.568849 7250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:30.571970 7081 master_runner.cc:428] Detected that this master 9c3bdafc02074b0f949e37e84091d49f is joining an existing cluster
I20260929 14:42:30.572031 7081 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44347
I20260929 14:42:30.583307 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.98.62:40905)
0929 14:42:30.574464 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.574468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.575243 (+ 775us) server_negotiation.cc:407] Connection header received
0929 14:42:30.575393 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.575396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.575446 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.575515 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.576590 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577350 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578534 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.578848 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578922 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.580380 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.580384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.580387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.580414 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.581806 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.582186 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.582190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.582192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.582280 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.582519 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.582522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.582524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.582707 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.582793 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.583039 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.583147 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:42:30.583324 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44838)
0929 14:42:30.575162 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.575173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.575275 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.575630 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.575633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.576367 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.576460 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.576467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577473 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.577476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.578420 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.578428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.579064 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.579067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.579442 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.579921 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.579935 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.580236 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.581910 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.581913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.581916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.582100 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.582360 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.582363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.582365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.582444 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.582798 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.582801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.582892 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.582974 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.583123 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":62,"thread_start_us":103,"threads_started":1}
I20260929 14:42:30.585210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6575
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:42:30.592670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44371
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40905
--webserver_port=40351
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:42:30.602702 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36494)
0929 14:42:30.575554 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.575566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.575661 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.575869 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.575872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.576097 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.576215 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.576222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577230 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.577233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.578516 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.578526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.579167 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.579170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.579253 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.597408 (+ 18155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.597425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.597925 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.601005 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.601011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.601233 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.601494 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.601499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.601573 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.602194 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.602197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.602269 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.602355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.602500 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":32,"thread_start_us":77,"threads_started":1}
I20260929 14:42:30.603291 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.0.98.60:44347)
0929 14:42:30.575079 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.575083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.575617 (+ 534us) server_negotiation.cc:407] Connection header received
0929 14:42:30.575723 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.575726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.575762 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.575832 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.576330 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577113 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578632 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.578984 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.579062 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.598092 (+ 19030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.598096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.598098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.598122 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.600899 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601332 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.601335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.601337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.601408 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601657 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.601660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.601662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.602098 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.602163 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.603042 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.603147 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":232,"server-negotiator.queue_time_us":42}
I20260929 14:42:30.603453 7239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:30.603626 7239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } }
I20260929 14:42:30.604390 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.587655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36500)
0929 14:42:30.587789 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.587804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.587883 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.588404 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.588407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.588601 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.588689 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.588694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.589711 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.589715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.590437 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.590447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.590927 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.590930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.591020 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.591493 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.591509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.592972 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.600833 (+ 7861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.600837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.600839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.601135 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.601450 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.601455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.601539 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.603825 (+ 2286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.603828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.603947 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.604053 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.604213 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 14:42:30.605187 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.587879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.98.60:44347)
0929 14:42:30.588142 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.588148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.588158 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:42:30.588233 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.588235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.588279 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.588356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.588803 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.589569 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.590560 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.590814 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.590898 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.593163 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.593168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.593171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.593202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.600377 (+ 7175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601249 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.601253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.601255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.601349 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.601631 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.601634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.601636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.601926 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.602009 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.604723 (+ 2714us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.604982 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":194,"thread_start_us":113,"threads_started":1}
I20260929 14:42:30.607231 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c3bdafc02074b0f949e37e84091d49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e506138610264abd9f8ce3034ffddafc" is_pre_election: true
I20260929 14:42:30.607825 6512 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44347: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44347
}
attrs {
promote: true
}
}
dest_uuid: "e506138610264abd9f8ce3034ffddafc"
cas_config_opid_index: 5
I20260929 14:42:30.607946 7239 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905), 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61:39265)
I20260929 14:42:30.608464 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c3bdafc02074b0f949e37e84091d49f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" is_pre_election: true
I20260929 14:42:30.608470 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } }
I20260929 14:42:30.608642 7125 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Handling vote request from an unknown peer 9c3bdafc02074b0f949e37e84091d49f
I20260929 14:42:30.608764 7125 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 9c3bdafc02074b0f949e37e84091d49f for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 14:42:30.609216 7181 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c3bdafc02074b0f949e37e84091d49f; no voters: 43ea92adb9fa4235b9277ad0ee5d961b, e506138610264abd9f8ce3034ffddafc
I20260929 14:42:30.609730 7239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:30.609792 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.610983 7239 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:42:30.611202 7075 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44347
I20260929 14:42:30.611372 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.611460 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:30.611882 7081 master.cc:537] Master@127.0.98.60:44347 shutting down...
I20260929 14:42:30.611936 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.98.61:39265)
0929 14:42:30.574780 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.574784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.575295 (+ 511us) server_negotiation.cc:407] Connection header received
0929 14:42:30.575450 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.575453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.575501 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.575564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.576227 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577043 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578204 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.578550 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578617 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.598543 (+ 19926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.598547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.598550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.598589 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.600528 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.602163 (+ 1635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.602167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.602169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.602278 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.602593 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.602597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.602598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.603119 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.611175 (+ 8056us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.611609 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.611762 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:42:30.612511 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.574608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33980)
0929 14:42:30.575214 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.575228 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.575329 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.575652 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.575654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.575893 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.576057 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.576102 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.577164 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.577167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.578077 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.578087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.578783 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.578786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.578925 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.579424 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.579440 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.598344 (+ 18904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.600683 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.600686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.600689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.601986 (+ 1297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.602405 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.602408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.602410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.602492 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.611278 (+ 8786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.611281 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.611415 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.612138 (+ 723us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.612316 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":20,"thread_start_us":75,"threads_started":1}
W20260929 14:42:30.616451 6497 consensus_peers.cc:487] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc -> Peer 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60:44347): Couldn't send request to peer 9c3bdafc02074b0f949e37e84091d49f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:42:30.617365 7081 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:30.617666 7081 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:30.617774 7081 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: stopping tablet replica
I20260929 14:42:30.621305 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.613934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:33984)
0929 14:42:30.615409 (+ 1475us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.615424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.615503 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.616713 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.616716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.616739 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:30.616915 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.616922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.618334 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.618338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.619671 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.619684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.620765 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.620781 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.620938 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.621004 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.621083 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.621168 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394}
I20260929 14:42:30.622128 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.614140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.98.61:39265)
0929 14:42:30.615647 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.615651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.615663 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:30.615753 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.615756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.615823 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.615907 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:30.617043 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.618080 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.619834 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.620445 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.620545 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.620698 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.621933 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.622015 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434}
I20260929 14:42:30.622717 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Refusing update from remote peer e506138610264abd9f8ce3034ffddafc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 14:42:30.622931 7081 master.cc:552] Master@127.0.98.60:44347 shutdown complete.
I20260929 14:42:30.622970 7081 master_runner.cc:305] Clearing existing system tablet
I20260929 14:42:30.623195 7081 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.623312 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:42:30.623667 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.625133 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.625249 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.625880 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.626122 7081 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.626221 7081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.626250 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "9c3bdafc02074b0f949e37e84091d49f"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "854cc38423372c5b863b648cea9a4efa"
server_key_iv: "92f1e3948ade88a808b2748fbdedde0c"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.627382 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e506138610264abd9f8ce3034ffddafc. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.627498 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: LEARNER
I20260929 14:42:30.627588 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: true } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.627646 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: LEARNER
I20260929 14:42:30.628324 7081 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:30.629540 7074 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: attempting to promote NON_VOTER 43ea92adb9fa4235b9277ad0ee5d961b to VOTER
I20260929 14:42:30.629980 7074 consensus_queue.cc:237] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } }
I20260929 14:42:30.630620 7081 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 14:42:30.630690 7081 log.cc:1196] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 14:42:30.630923 7081 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Deleting consensus metadata
I20260929 14:42:30.631211 7081 master_runner.cc:315] Copying system tablet from 127.0.98.62:40905
I20260929 14:42:30.631919 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 LEARNER]: Refusing update from remote peer e506138610264abd9f8ce3034ffddafc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 14:42:30.632297 7081 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40905
I20260929 14:42:30.632386 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.631430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36516)
0929 14:42:30.632222 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.632303 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":710}
I20260929 14:42:30.633116 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:42:30.633862 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.633588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36522)
0929 14:42:30.633724 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.633788 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 14:42:30.634459 6497 consensus_peers.cc:487] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc -> Peer 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60:44347): Couldn't send request to peer 9c3bdafc02074b0f949e37e84091d49f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:42:30.635399 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.635896 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.636940 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.637043 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:30.637147 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.637205 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:30.637617 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.637758 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:30.646303 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.633034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.0.98.62:40905)
0929 14:42:30.633135 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.633139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.635128 (+ 1989us) server_negotiation.cc:407] Connection header received
0929 14:42:30.638386 (+ 3258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.638390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.638448 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.638534 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.639066 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.639905 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.641080 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.641427 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.641535 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.642833 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.642837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.642841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.642869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.644480 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.645035 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.645039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.645041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.645131 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.645412 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.645415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.645417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.645642 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.645741 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.646024 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.646146 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:30.646311 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.633240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:44842)
0929 14:42:30.635043 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.635058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.638121 (+ 3063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.638578 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.638581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.638821 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.638953 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.638961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.640029 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.640032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.640962 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.640974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.641556 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.641560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.641659 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.642244 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.642258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.642682 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.644606 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.644610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.644613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.644922 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.645222 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.645225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.645227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.645321 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.645732 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.645735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.645849 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.645946 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.646104 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":94,"threads_started":1}
I20260929 14:42:30.647034 6533 tablet_copy_service.cc:140] P e506138610264abd9f8ce3034ffddafc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9c3bdafc02074b0f949e37e84091d49f ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44842)
I20260929 14:42:30.647131 6533 tablet_copy_service.cc:161] P e506138610264abd9f8ce3034ffddafc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9c3bdafc02074b0f949e37e84091d49f at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44842: session id = 9c3bdafc02074b0f949e37e84091d49f-00000000000000000000000000000000
I20260929 14:42:30.648224 6533 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:42:30.649610 7081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 14:42:30.652196 7081 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: tablet copy: Starting download of 0 data blocks...
I20260929 14:42:30.652405 7081 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: tablet copy: Starting download of 1 WAL segments...
I20260929 14:42:30.654163 7081 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:42:30.656275 6533 tablet_copy_service.cc:342] P e506138610264abd9f8ce3034ffddafc: Request end of tablet copy session 9c3bdafc02074b0f949e37e84091d49f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44842
I20260929 14:42:30.656373 6533 tablet_copy_service.cc:433] P e506138610264abd9f8ce3034ffddafc: ending tablet copy session 9c3bdafc02074b0f949e37e84091d49f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9c3bdafc02074b0f949e37e84091d49f
I20260929 14:42:30.664438 7081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.666591 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.666829 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.666611 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.669047 7081 server_base.cc:1034] running on GCE node
I20260929 14:42:30.669235 7081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.669562 7081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.670704 7081 hybrid_clock.cc:648] HybridClock initialized: now 1790692950670683 us; error 39 us; skew 500 ppm
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.672319 7081 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 14:42:30.672827 7081 webserver.cc:460] Webserver started at http://127.0.98.60:39211/ using document root <none> and password file <none>
I20260929 14:42:30.673071 7081 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.673115 7081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.674448 7081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.675455 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.675726 7081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:30.675810 7081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "9c3bdafc02074b0f949e37e84091d49f"
format_stamp: "Formatted at 2026-09-29 14:42:30 on dist-test-slave-g8hp"
server_key: "854cc38423372c5b863b648cea9a4efa"
server_key_iv: "92f1e3948ade88a808b2748fbdedde0c"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.675894 7081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.695655 7081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.698076 7081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.703118 7081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44347
I20260929 14:42:30.703162 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44347 every 8 connection(s)
I20260929 14:42:30.703758 7081 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:42:30.705649 7334 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:42:30.706840 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Bootstrap starting.
I20260929 14:42:30.719084 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:42:30.719379 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: Bootstrap complete.
I20260929 14:42:30.719978 7334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } }
I20260929 14:42:30.720117 7334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c3bdafc02074b0f949e37e84091d49f, State: Initialized, Role: LEARNER
I20260929 14:42:30.720206 7334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } }
I20260929 14:42:30.720654 7334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:30.724232 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:30.724386 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: LEARNER
I20260929 14:42:30.725744 7348 catalog_manager.cc:1259] Loaded cluster ID: 62c6a16bba614ba7a14d79c04bba8b0f
I20260929 14:42:30.725816 7348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: loading cluster ID for follower catalog manager: success
I20260929 14:42:30.727967 7348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: acquiring CA information for follower catalog manager: success
I20260929 14:42:30.729548 7348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:42:30.739416 6838 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
W20260929 14:42:30.740360 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:30.740626 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:30.740694 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:30.744947 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:30.745016 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:30.745079 7260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:30.750466 7260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40351
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:30.751581 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:30.752635 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.762879 7356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.762882 7353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.762882 7354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.763296 7260 server_base.cc:1034] running on GCE node
I20260929 14:42:30.763797 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.764514 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.765707 7260 hybrid_clock.cc:648] HybridClock initialized: now 1790692950765677 us; error 38 us; skew 500 ppm
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.768642 7260 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:42:30.769584 7260 webserver.cc:460] Webserver started at http://127.0.98.1:40351/ using document root <none> and password file <none>
I20260929 14:42:30.770157 7260 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.770341 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.774389 7260 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 14:42:30.776525 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.777940 7260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:30.778052 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "91aae8b7928041d9876cfc190512f8e8"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "535e377249ec059c0356671e2969babd"
server_key_iv: "37c9444320ce947650344912dd57f191"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.778600 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.790319 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.793627 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:30.793818 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.794418 7260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:30.795275 7260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:30.795321 7260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.795373 7260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:30.795403 7260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.804121 7260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44371
I20260929 14:42:30.804184 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44371 every 8 connection(s)
I20260929 14:42:30.805192 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:30.812556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7260
I20260929 14:42:30.812762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6709
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:30.820806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39919
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40905
--webserver_port=44931
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:42:30.821473 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.808637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46121 (local address 127.0.98.61:39265)
0929 14:42:30.808767 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.808772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.809112 (+ 340us) server_negotiation.cc:407] Connection header received
0929 14:42:30.809404 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.809407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.809457 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.809524 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.811043 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811828 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.813719 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.814043 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.814123 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.817531 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.817535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.817538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.817568 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.819079 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819638 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.819641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.819643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.819722 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.820150 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.820154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.820156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.820389 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.820471 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.821180 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.821298 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:42:30.821645 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.806872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42555 (local address 127.0.98.60:44347)
0929 14:42:30.807104 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.807114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.808184 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 14:42:30.809297 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.809301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.809358 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.809443 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.811041 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811751 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.813461 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.813819 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.813893 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.817849 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.817853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.817855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.817884 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.819341 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819784 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.819787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.819789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.819875 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.820181 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.820184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.820186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.820386 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.820468 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.821332 (+ 864us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.821459 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260929 14:42:30.823028 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.808962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47723 (local address 127.0.98.62:40905)
0929 14:42:30.809082 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:30.809087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:30.809103 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:42:30.809578 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:30.809581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:30.809630 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:30.809698 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:30.811030 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811834 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.813807 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.814149 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.814220 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.819512 (+ 5292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:30.819516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:30.819519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:30.819548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:30.821180 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.821727 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.821731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.821733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.821820 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:30.822097 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:30.822100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:30.822102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:30.822304 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:30.822379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:30.822785 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 14:42:30.822899 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:42:30.823190 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.808466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.98.1:46121)
0929 14:42:30.809039 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.809050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.809160 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.809605 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.809614 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.810153 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.810887 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.810897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811930 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.811933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.813591 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.813602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.814273 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.814276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.814361 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.815192 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.815205 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.817358 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.819190 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819196 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.819214 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.819531 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.819841 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.819846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.820030 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.820541 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.820544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.820893 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.821624 (+ 731us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.822070 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":378,"spinlock_wait_cycles":7296,"thread_start_us":70,"threads_started":1}
I20260929 14:42:30.823508 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.808778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:47723)
0929 14:42:30.809019 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.809032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.809129 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.809736 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.809739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.810116 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.810878 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.810893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811970 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.811973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.813702 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.813709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.814433 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.814436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.814522 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.815080 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.815094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.818889 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.821298 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.821301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.821303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.821619 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.821919 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.821922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.821924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.822000 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.822451 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.822454 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.822605 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.823214 (+ 609us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.823355 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":230,"thread_start_us":99,"threads_started":1}
I20260929 14:42:30.823750 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:30.807215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.98.1:42555)
0929 14:42:30.808031 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:30.808072 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:30.809084 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:30.809605 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:30.809615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:30.810125 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:30.810891 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.810897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.811889 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.811892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.813334 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:30.813345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:30.814094 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:30.814098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:30.814301 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:30.814994 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:30.815014 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:30.817709 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:30.819441 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.819446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.819669 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.819974 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:30.819977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:30.819978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:30.820059 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:30.820537 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:30.820543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:30.821123 (+ 580us) client_negotiation.cc:769] Sending connection context
0929 14:42:30.822245 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0929 14:42:30.822397 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":163,"spinlock_wait_cycles":8960,"thread_start_us":516,"threads_started":1}
I20260929 14:42:30.824747 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
I20260929 14:42:30.824793 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39265
I20260929 14:42:30.824801 7478 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44347
I20260929 14:42:30.825001 7478 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:30.825001 7479 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:30.825001 7477 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:30.825618 7477 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:30.825618 7478 heartbeater.cc:511] Master 127.0.98.60:44347 requested a full tablet report, sending...
I20260929 14:42:30.825620 7479 heartbeater.cc:511] Master 127.0.98.61:39265 requested a full tablet report, sending...
I20260929 14:42:30.827127 6512 ts_manager.cc:194] Re-registered known tserver with Master: 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371)
I20260929 14:42:30.827340 7107 ts_manager.cc:194] Registered new tserver with Master: 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371)
I20260929 14:42:30.827886 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47723
I20260929 14:42:30.827988 7298 ts_manager.cc:194] Registered new tserver with Master: 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371)
W20260929 14:42:30.936962 7485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:30.937228 7485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:30.937283 7485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:30.941287 7485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:30.941339 7485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:30.941402 7485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:30.946691 7485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44931
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:30.947768 7485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:30.948889 7485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:30.958961 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.958932 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:30.958927 7493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:30.959300 7485 server_base.cc:1034] running on GCE node
I20260929 14:42:30.959699 7485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:30.960417 7485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:30.961604 7485 hybrid_clock.cc:648] HybridClock initialized: now 1790692950961563 us; error 49 us; skew 500 ppm
I20260929 14:42:30.961625 6972 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
Sep 29 14:42:30 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:30.964533 7485 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:42:30.965456 7485 webserver.cc:460] Webserver started at http://127.0.98.2:44931/ using document root <none> and password file <none>
I20260929 14:42:30.965999 7485 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:30.966073 7485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:30.970222 7485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:30.972381 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:30.973889 7485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:30.974015 7485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "0828b6e790f746eca979c0d54b186fed"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "66fa18cefe5f6cad1b8759aac6ba5c8f"
server_key_iv: "ada398be001761be8123db6561c78bc0"
server_key_version: "encryptionkey@0"
I20260929 14:42:30.974627 7485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:30.995225 7485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:30.998582 7485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:30.998754 7485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:30.999396 7485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:31.000275 7485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:31.000324 7485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:31.000366 7485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:31.000396 7485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:31.009305 7485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39919
I20260929 14:42:31.009398 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39919 every 8 connection(s)
I20260929 14:42:31.010367 7485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:31.012112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7485
I20260929 14:42:31.012301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6843
I20260929 14:42:31.019737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33323
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40905
--webserver_port=42861
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:42:31.029325 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.014713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44795 (local address 127.0.98.62:40905)
0929 14:42:31.014839 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.014843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.015177 (+ 334us) server_negotiation.cc:407] Connection header received
0929 14:42:31.015747 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.015750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.015812 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.015860 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.017761 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018579 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020405 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.020729 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020811 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.024699 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.024703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.024705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.024734 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.026808 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027417 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.027505 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027895 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.028115 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.028208 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.029019 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.029150 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:31.029325 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.012200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35015 (local address 127.0.98.60:44347)
0929 14:42:31.012420 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.012425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.014616 (+ 2191us) server_negotiation.cc:407] Connection header received
0929 14:42:31.015747 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.015750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.015806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.015858 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.017759 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018579 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020455 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.020799 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020930 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.025241 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.025245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.025247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.025286 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.026802 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027416 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.027505 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027894 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.028115 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.028212 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.029019 (+ 807us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.029150 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 14:42:31.029325 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.014158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54379 (local address 127.0.98.61:39265)
0929 14:42:31.014336 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.014340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.014596 (+ 256us) server_negotiation.cc:407] Connection header received
0929 14:42:31.015834 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.015838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.015887 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.015957 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.017765 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018501 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020427 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.020744 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.020816 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.024415 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.024419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.024423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.024454 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.026637 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027384 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.027481 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027912 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.027915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.027917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.028161 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.028239 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.029029 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.029159 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 14:42:31.031113 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.014052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.98.2:54379)
0929 14:42:31.014447 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.014501 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.015516 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.016180 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.016189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.016807 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.017591 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.017611 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018679 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.018682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.020267 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.020277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.021046 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.021050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.021337 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.022110 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.022133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.024197 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.026789 (+ 2592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.026796 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.026815 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027260 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.027594 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.027600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027801 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.028361 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.028364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.028755 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.029581 (+ 826us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.029981 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":341,"spinlock_wait_cycles":25216,"thread_start_us":74,"threads_started":1}
I20260929 14:42:31.031111 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.012653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.98.2:35015)
0929 14:42:31.014447 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.014545 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.015507 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.016179 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.016189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.016780 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.017595 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.017611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018718 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.018721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.020313 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.020320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.021071 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.021075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.021362 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.022246 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.022259 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.025125 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.026923 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.026926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.026928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027304 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.027606 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.027611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027790 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.028284 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.028287 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.028689 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.029586 (+ 897us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.030055 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1571,"mutex_wait_us":500,"spinlock_wait_cycles":65408,"thread_start_us":581,"threads_started":1}
I20260929 14:42:31.031649 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.014385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:44795)
0929 14:42:31.015080 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.015092 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.015509 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.016179 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.016189 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.016807 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.017592 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.017611 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.018718 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.018721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.020266 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.020276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.021046 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.021050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.021348 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.022147 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.022162 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.024536 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.026923 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.026926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.026928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027304 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.027606 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.027609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.027611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.027799 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.028275 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.028282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.028689 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.030015 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.030129 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":484,"spinlock_wait_cycles":85376,"thread_start_us":69,"threads_started":1}
I20260929 14:42:31.032390 7615 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44347
I20260929 14:42:31.032459 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39265
I20260929 14:42:31.032621 7616 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.032624 7615 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.033226 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
I20260929 14:42:31.033403 7614 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.033546 7614 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:31.033224 7616 heartbeater.cc:511] Master 127.0.98.61:39265 requested a full tablet report, sending...
I20260929 14:42:31.033242 7615 heartbeater.cc:511] Master 127.0.98.60:44347 requested a full tablet report, sending...
I20260929 14:42:31.034564 6512 ts_manager.cc:194] Re-registered known tserver with Master: 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
I20260929 14:42:31.034708 7298 ts_manager.cc:194] Registered new tserver with Master: 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
I20260929 14:42:31.035311 6512 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44795
I20260929 14:42:31.035835 7107 ts_manager.cc:194] Registered new tserver with Master: 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
W20260929 14:42:31.136811 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:31.137158 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:31.137218 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:31.141502 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:31.141569 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:31.141657 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:31.146996 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42861
--tserver_master_addrs=127.0.98.62:40905,127.0.98.61:39265,127.0.98.60:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:31.148046 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:31.149124 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:31.159318 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:31.159318 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:31.159412 7621 server_base.cc:1034] running on GCE node
W20260929 14:42:31.159368 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:31.159989 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:31.160688 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:31.161850 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790692951161830 us; error 59 us; skew 500 ppm
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692951, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:31.164654 7621 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:42:31.165613 7621 webserver.cc:460] Webserver started at http://127.0.98.3:42861/ using document root <none> and password file <none>
I20260929 14:42:31.166203 7621 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:31.166350 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:31.170477 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:31.172508 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:31.174018 7621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:31.174153 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0cbe9e1104fb4153ad340681d9da5760"
format_stamp: "Formatted at 2026-09-29 14:42:29 on dist-test-slave-g8hp"
server_key: "8756ce17eab5f372a607441022d6aaa8"
server_key_iv: "41ead26aae190f2f0d93c901be75a318"
server_key_version: "encryptionkey@0"
I20260929 14:42:31.174706 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:31.197208 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:31.199851 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.195129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:36536)
0929 14:42:31.195417 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.195431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.195544 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.195856 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.195859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.195881 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.195991 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.195998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.197106 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.197110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.198187 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.198199 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.199378 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.199383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.199483 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.199512 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.199591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.199693 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":71,"threads_started":1}
I20260929 14:42:31.200176 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.195214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.98.60:44347)
0929 14:42:31.195327 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.195331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.195499 (+ 168us) server_negotiation.cc:407] Connection header received
0929 14:42:31.195669 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.195672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.195735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.195829 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.196098 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.196973 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.198322 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.199242 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.199341 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.199497 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.199609 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.199680 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:42:31.200892 7303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 14:42:31.201102 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:31.201296 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:31.201315 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e506138610264abd9f8ce3034ffddafc. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:31.201444 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: LEARNER
I20260929 14:42:31.202055 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:31.203065 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:31.203114 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:31.203176 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:31.203210 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:31.211752 7621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33323
I20260929 14:42:31.211822 7750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33323 every 8 connection(s)
I20260929 14:42:31.212808 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:31.216847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7621
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692951, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692951, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 29 14:42:31 dist-test-slave-g8hp krb5kdc[6468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692951, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 14:42:31.229768 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.219938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.0.98.62:40905)
0929 14:42:31.220198 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.220202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.220503 (+ 301us) server_negotiation.cc:407] Connection header received
0929 14:42:31.220612 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.220615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.220663 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.220746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.221698 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.222436 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.223639 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.223950 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.224034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.225693 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.225697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.225700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.225727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.227346 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.227794 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.227798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.227799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.227897 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.228257 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.228260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.228262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.228550 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.228675 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.229406 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.229582 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260929 14:42:31.230840 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.214736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54095 (local address 127.0.98.62:40905)
0929 14:42:31.214881 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.214885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.215755 (+ 870us) server_negotiation.cc:407] Connection header received
0929 14:42:31.216988 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.216992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.217047 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.217102 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.218543 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.219334 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.221990 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.222365 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.222432 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.226927 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.226931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.226934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.226959 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.228409 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.229064 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.229067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.229069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.229149 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.229553 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.229574 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.229575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.229784 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.229857 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.230575 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.230698 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":39}
I20260929 14:42:31.231516 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.220154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.0.98.61:39265)
0929 14:42:31.220374 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.220378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.220900 (+ 522us) server_negotiation.cc:407] Connection header received
0929 14:42:31.221007 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.221009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.221054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.221676 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.222507 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.223264 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.224368 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.224708 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.224802 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.226491 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.226496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.226499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.226524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.229631 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.230048 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.230051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.230052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.230148 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.230420 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.230423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.230425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.230632 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.230744 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.231253 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.231368 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260929 14:42:31.232438 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.215169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:54095)
0929 14:42:31.215602 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.215645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.216612 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.217289 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.217297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.217733 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.218365 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.218375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.219577 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.219580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.221878 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.221887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.222590 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.222594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.223831 (+ 1237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.224210 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.224223 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.226767 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.228552 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.228559 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.228578 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.228950 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.229264 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.229267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.229269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.229446 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.229949 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.229956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.230300 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.231041 (+ 741us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.231392 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":53,"spinlock_wait_cycles":5376,"thread_start_us":120,"threads_started":1}
I20260929 14:42:31.233445 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.216218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35469 (local address 127.0.98.60:44347)
0929 14:42:31.216339 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.216344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.216471 (+ 127us) server_negotiation.cc:407] Connection header received
0929 14:42:31.218900 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.218903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.218958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.219014 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.219896 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.220589 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.221879 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.222214 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.222290 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.227203 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.227206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.227209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.227237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.229745 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.230221 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.230224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.230226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.230310 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.231942 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.231945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.231947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.232308 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.232386 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.233175 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.233294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:42:31.233543 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40905
I20260929 14:42:31.233757 7751 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.234203 7751 heartbeater.cc:511] Master 127.0.98.62:40905 requested a full tablet report, sending...
I20260929 14:42:31.234251 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.216084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.98.3:35469)
0929 14:42:31.216377 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.216392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.218767 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.219154 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.219157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.219339 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.219450 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.219458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.220785 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.220788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.221697 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.221706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.222520 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.222524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.222773 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.223398 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.223420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.227055 (+ 3635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.229888 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.229892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.229894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.230112 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.230412 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.230415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.230417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.230510 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.232442 (+ 1932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.232445 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.232536 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.233608 (+ 1072us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.233765 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260929 14:42:31.234889 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44347
I20260929 14:42:31.234962 7752 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.235108 7752 heartbeater.cc:511] Master 127.0.98.60:44347 requested a full tablet report, sending...
I20260929 14:42:31.235150 6513 ts_manager.cc:194] Re-registered known tserver with Master: 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
I20260929 14:42:31.235759 7298 ts_manager.cc:194] Registered new tserver with Master: 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
I20260929 14:42:31.235772 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54095
I20260929 14:42:31.235924 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.216742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49721 (local address 127.0.98.61:39265)
0929 14:42:31.216900 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.216904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.216917 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:31.216957 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.216959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.217005 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.217076 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.218503 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.219320 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.222896 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.223227 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.223301 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.227572 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.227583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.227586 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.227611 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.229497 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.232690 (+ 3193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.232693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.232695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.232799 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.234538 (+ 1739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.234542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.234544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.234759 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.234837 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.235646 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.235778 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:42:31.236373 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.216403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.98.3:49721)
0929 14:42:31.216614 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.216627 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.216726 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.217288 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.217295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.217722 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:31.218350 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.218364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.219482 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.219486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.221517 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.221527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.223455 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.223458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.223539 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.224030 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:31.224043 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:31.226662 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:31.230613 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.230615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.230617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.230825 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.232918 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:31.232921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:31.232922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:31.234427 (+ 1505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:31.235377 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:31.235380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:31.235472 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.236042 (+ 570us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.236201 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":824,"thread_start_us":71,"threads_started":1}
I20260929 14:42:31.237030 7753 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39265
I20260929 14:42:31.237119 7753 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:31.237255 7753 heartbeater.cc:511] Master 127.0.98.61:39265 requested a full tablet report, sending...
I20260929 14:42:31.233341 6512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:42:31.237879 7107 ts_manager.cc:194] Registered new tserver with Master: 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
W20260929 14:42:31.238670 6512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:31.241302 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: true } } }
I20260929 14:42:31.241433 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:31.243631 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.219666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.0.98.60:44347)
0929 14:42:31.219910 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.219920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.225609 (+ 5689us) server_negotiation.cc:407] Connection header received
0929 14:42:31.225793 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.225797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.225850 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.225909 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.236571 (+ 10662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.237337 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.238527 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.238770 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.238865 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.240362 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.240366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.240369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.240395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.242014 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.242449 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.242453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.242455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.242532 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.242840 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.242843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.242845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.243024 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.243133 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.243363 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.243479 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260929 14:42:31.244552 7770 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: attempting to promote NON_VOTER 9c3bdafc02074b0f949e37e84091d49f to VOTER
I20260929 14:42:31.244903 7770 consensus_queue.cc:237] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } }
I20260929 14:42:31.246158 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Refusing update from remote peer e506138610264abd9f8ce3034ffddafc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:42:31.246330 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 LEARNER]: Refusing update from remote peer e506138610264abd9f8ce3034ffddafc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 14:42:31.246459 7769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:42:31.246742 7769 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:42:31.248088 7770 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.248413 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.249317 7303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.249876 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.249971 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:31.250131 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.250203 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:31.250381 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.250452 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:31.251516 7337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.251623 7337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:31.252286 7778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e506138610264abd9f8ce3034ffddafc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:31.252422 7778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:31.261720 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.256001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33323 (local address 127.0.0.1:45830)
0929 14:42:31.256128 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.256144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.256215 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.257063 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.257066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.257082 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.257201 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.257222 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.259014 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.259016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.260101 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.260111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.261316 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.261319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.261404 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.261426 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.261499 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.261575 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 14:42:31.261875 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.256170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.3:33323)
0929 14:42:31.256504 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.256515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.256539 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:42:31.256618 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.256625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.256906 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.257085 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.257314 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.258936 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.260251 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.261201 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.261302 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.261453 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.261640 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.261740 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":117,"threads_started":1}
I20260929 14:42:31.262355 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.256275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.0.1:56086)
0929 14:42:31.256794 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.256805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.256906 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.258148 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.258151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.258168 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.258274 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.258281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.259330 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.259334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.260381 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.260389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.261784 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.261787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.262067 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.262089 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.262156 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.262225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 14:42:31.262430 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.257186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.0.98.1:44371)
0929 14:42:31.257549 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.257561 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.257582 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:42:31.257772 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.257778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.258029 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.258178 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.258356 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.259226 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.260707 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.261501 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.261578 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.261698 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.262205 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.262301 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":174,"threads_started":1}
I20260929 14:42:31.264192 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.256579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39919 (local address 127.0.0.1:45148)
0929 14:42:31.258791 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.258805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.258880 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.259617 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.259620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.259639 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.259737 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.259743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.261097 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.261101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.262343 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.262352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.263824 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.263827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.263911 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.263934 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.263992 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.264059 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"mutex_wait_us":20,"thread_start_us":84,"threads_started":1}
I20260929 14:42:31.264673 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.257126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.0.98.2:39919)
0929 14:42:31.259184 (+ 2058us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.259195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.259217 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:42:31.259287 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.259293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.259483 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.259629 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.259846 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.260993 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.262598 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.263185 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.263295 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.264389 (+ 1094us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.264447 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.264542 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1959,"thread_start_us":102,"threads_started":1}
I20260929 14:42:31.264719 7685 tablet_service.cc:1467] Processing CreateTablet for tablet a490d145704f4488b39eac0a5c55e346 (DEFAULT_TABLE table=test-table [id=ed11d0a691b446b68224350a0cf6403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:31.265218 7411 tablet_service.cc:1467] Processing CreateTablet for tablet a490d145704f4488b39eac0a5c55e346 (DEFAULT_TABLE table=test-table [id=ed11d0a691b446b68224350a0cf6403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:31.265769 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a490d145704f4488b39eac0a5c55e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:31.266047 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a490d145704f4488b39eac0a5c55e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:31.270931 7548 tablet_service.cc:1467] Processing CreateTablet for tablet a490d145704f4488b39eac0a5c55e346 (DEFAULT_TABLE table=test-table [id=ed11d0a691b446b68224350a0cf6403a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:31.272078 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a490d145704f4488b39eac0a5c55e346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:31.273550 7785 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Bootstrap starting.
I20260929 14:42:31.273972 7784 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Bootstrap starting.
I20260929 14:42:31.275959 7784 tablet_bootstrap.cc:654] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:31.276713 7784 log.cc:824] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:31.278667 7784 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: No bootstrap required, opened a new log
I20260929 14:42:31.278880 7784 ts_tablet_manager.cc:1397] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:42:31.279693 7785 tablet_bootstrap.cc:654] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:31.280048 7787 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Bootstrap starting.
I20260929 14:42:31.280743 7785 log.cc:824] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:31.282423 7787 tablet_bootstrap.cc:654] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:31.282603 7785 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: No bootstrap required, opened a new log
I20260929 14:42:31.282781 7785 ts_tablet_manager.cc:1397] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 14:42:31.283252 7787 log.cc:824] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:31.283195 7784 raft_consensus.cc:348] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.283483 7784 raft_consensus.cc:374] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:31.283542 7784 raft_consensus.cc:729] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91aae8b7928041d9876cfc190512f8e8, State: Initialized, Role: FOLLOWER
I20260929 14:42:31.283972 7784 consensus_queue.cc:260] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.284588 7477 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
I20260929 14:42:31.284758 7784 ts_tablet_manager.cc:1428] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260929 14:42:31.286990 7785 raft_consensus.cc:348] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.287278 7785 raft_consensus.cc:374] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:31.287333 7785 raft_consensus.cc:729] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbe9e1104fb4153ad340681d9da5760, State: Initialized, Role: FOLLOWER
I20260929 14:42:31.287812 7785 consensus_queue.cc:260] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.288897 7785 ts_tablet_manager.cc:1428] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:42:31.289261 7751 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
I20260929 14:42:31.291392 7787 tablet_bootstrap.cc:492] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: No bootstrap required, opened a new log
I20260929 14:42:31.291666 7787 ts_tablet_manager.cc:1397] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20260929 14:42:31.295305 7787 raft_consensus.cc:348] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.295562 7787 raft_consensus.cc:374] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:31.295603 7787 raft_consensus.cc:729] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0828b6e790f746eca979c0d54b186fed, State: Initialized, Role: FOLLOWER
I20260929 14:42:31.296057 7787 consensus_queue.cc:260] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.296808 7614 heartbeater.cc:503] Master 127.0.98.62:40905 was elected leader, sending a full tablet report...
I20260929 14:42:31.297195 7787 ts_tablet_manager.cc:1428] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260929 14:42:31.307057 7480 tablet.cc:2369] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:31.453256 7791 raft_consensus.cc:482] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:31.453503 7791 raft_consensus.cc:504] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.454651 7791 leader_election.cc:288] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371), 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
I20260929 14:42:31.458461 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.455151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39919 (local address 127.0.98.3:50545)
0929 14:42:31.455286 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.455297 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.455375 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.455598 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.455601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.455617 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.455722 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.455730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.456482 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.456485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.457389 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.457414 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.458038 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.458041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.458135 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.458165 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.458244 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.458304 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":46}
I20260929 14:42:31.458464 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.455267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50545 (local address 127.0.98.2:39919)
0929 14:42:31.455405 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.455409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.455423 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:31.455470 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.455473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.455516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.455574 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.455833 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.456383 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.457501 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.457948 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.458028 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.458101 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.458270 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.458344 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:42:31.458490 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.454900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.98.3:60503)
0929 14:42:31.455063 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:31.455079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:31.455187 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:31.455533 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:31.455536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:31.455552 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:31.455669 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.455675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.456404 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.456407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.457386 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:31.457395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.458142 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.458145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:31.458241 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.458262 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:42:31.458332 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:42:31.458390 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":28}
I20260929 14:42:31.458531 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.455026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60503 (local address 127.0.98.1:44371)
0929 14:42:31.455162 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.455166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.455190 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:42:31.455395 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.455398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.455441 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.455504 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:31.455809 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.456299 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.457505 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.458013 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.458133 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.458192 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.458358 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.458428 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:42:31.459153 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a490d145704f4488b39eac0a5c55e346" candidate_uuid: "0cbe9e1104fb4153ad340681d9da5760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0828b6e790f746eca979c0d54b186fed" is_pre_election: true
I20260929 14:42:31.459534 7568 raft_consensus.cc:2454] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cbe9e1104fb4153ad340681d9da5760 in term 0.
I20260929 14:42:31.459996 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a490d145704f4488b39eac0a5c55e346" candidate_uuid: "0cbe9e1104fb4153ad340681d9da5760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91aae8b7928041d9876cfc190512f8e8" is_pre_election: true
I20260929 14:42:31.459995 7637 leader_election.cc:302] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0828b6e790f746eca979c0d54b186fed, 0cbe9e1104fb4153ad340681d9da5760; no voters:
I20260929 14:42:31.460282 7431 raft_consensus.cc:2454] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cbe9e1104fb4153ad340681d9da5760 in term 0.
I20260929 14:42:31.460340 7791 raft_consensus.cc:2781] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:31.460436 7791 raft_consensus.cc:482] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:31.460489 7791 raft_consensus.cc:3037] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:31.461627 7791 raft_consensus.cc:504] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.461927 7791 leader_election.cc:288] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [CANDIDATE]: Term 1 election: Requested vote from peers 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371), 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919)
I20260929 14:42:31.462288 7431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a490d145704f4488b39eac0a5c55e346" candidate_uuid: "0cbe9e1104fb4153ad340681d9da5760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91aae8b7928041d9876cfc190512f8e8"
I20260929 14:42:31.462396 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a490d145704f4488b39eac0a5c55e346" candidate_uuid: "0cbe9e1104fb4153ad340681d9da5760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0828b6e790f746eca979c0d54b186fed"
I20260929 14:42:31.462421 7431 raft_consensus.cc:3037] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:31.462517 7568 raft_consensus.cc:3037] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:31.463657 7568 raft_consensus.cc:2454] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cbe9e1104fb4153ad340681d9da5760 in term 1.
I20260929 14:42:31.463656 7431 raft_consensus.cc:2454] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cbe9e1104fb4153ad340681d9da5760 in term 1.
I20260929 14:42:31.463927 7637 leader_election.cc:302] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0828b6e790f746eca979c0d54b186fed, 0cbe9e1104fb4153ad340681d9da5760; no voters:
I20260929 14:42:31.464097 7791 raft_consensus.cc:2781] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:31.464304 7791 raft_consensus.cc:686] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 LEADER]: Becoming Leader. State: Replica: 0cbe9e1104fb4153ad340681d9da5760, State: Running, Role: LEADER
W20260929 14:42:31.464778 7754 tablet.cc:2369] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:31.464885 7791 consensus_queue.cc:237] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } }
I20260929 14:42:31.468401 6513 catalog_manager.cc:5495] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0cbe9e1104fb4153ad340681d9da5760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:31.500748 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.496998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.0.98.3:33323)
0929 14:42:31.497123 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.497126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.497219 (+ 93us) server_negotiation.cc:407] Connection header received
0929 14:42:31.497295 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.497297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.497351 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.497436 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:31.497683 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.498270 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.499078 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.499362 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.499442 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.499766 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:31.500277 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:31.500381 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.500535 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.500599 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:42:31.506814 7568 raft_consensus.cc:1261] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Refusing update from remote peer 0cbe9e1104fb4153ad340681d9da5760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:31.506816 7431 raft_consensus.cc:1261] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Refusing update from remote peer 0cbe9e1104fb4153ad340681d9da5760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:31.507522 7793 consensus_queue.cc:1035] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:31.507701 7791 consensus_queue.cc:1035] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:42:31.512106 7617 tablet.cc:2369] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:31.527750 6513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:31.527952 6513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:31.534694 6513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e506138610264abd9f8ce3034ffddafc: Sending DeleteTablet for 3 replicas of tablet a490d145704f4488b39eac0a5c55e346
I20260929 14:42:31.535718 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet a490d145704f4488b39eac0a5c55e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45830
I20260929 14:42:31.536116 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet a490d145704f4488b39eac0a5c55e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260929 14:42:31.536283 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet a490d145704f4488b39eac0a5c55e346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56086
I20260929 14:42:31.537608 7807 tablet_replica.cc:330] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: stopping tablet replica
I20260929 14:42:31.537884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6481
I20260929 14:42:31.541083 7807 raft_consensus.cc:2229] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:31.541487 7807 raft_consensus.cc:2258] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:31.542279 7806 tablet_replica.cc:330] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: stopping tablet replica
I20260929 14:42:31.542613 7806 raft_consensus.cc:2229] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:31.543043 7806 raft_consensus.cc:2258] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:31.543427 7808 tablet_replica.cc:330] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: stopping tablet replica
I20260929 14:42:31.543725 7808 raft_consensus.cc:2229] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:31.544039 7808 raft_consensus.cc:2258] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:31.544750 7806 ts_tablet_manager.cc:1905] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:42:31.545147 7284 connection.cc:535] server connection from 127.0.0.1:36536 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:42:31.545470 7808 ts_tablet_manager.cc:1905] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:31.545641 7807 ts_tablet_manager.cc:1905] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:31.548137 7808 ts_tablet_manager.cc:1918] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:31.548250 7808 log.cc:1196] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/a490d145704f4488b39eac0a5c55e346
I20260929 14:42:31.548556 7808 ts_tablet_manager.cc:1939] T a490d145704f4488b39eac0a5c55e346 P 91aae8b7928041d9876cfc190512f8e8: Deleting consensus metadata
W20260929 14:42:31.549921 7367 connection.cc:441] server connection from 127.0.0.1:56086 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56086 (request call id 1) could send its response
I20260929 14:42:31.551530 7806 ts_tablet_manager.cc:1918] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:31.551636 7806 log.cc:1196] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/a490d145704f4488b39eac0a5c55e346
I20260929 14:42:31.551954 7806 ts_tablet_manager.cc:1939] T a490d145704f4488b39eac0a5c55e346 P 0cbe9e1104fb4153ad340681d9da5760: Deleting consensus metadata
I20260929 14:42:31.548187 7807 ts_tablet_manager.cc:1918] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:31.553167 7807 log.cc:1196] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/a490d145704f4488b39eac0a5c55e346
I20260929 14:42:31.553458 7807 ts_tablet_manager.cc:1939] T a490d145704f4488b39eac0a5c55e346 P 0828b6e790f746eca979c0d54b186fed: Deleting consensus metadata
W20260929 14:42:31.553670 7637 connection.cc:441] server connection from 127.0.0.1:45830 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45830 (request call id 1) could send its response
W20260929 14:42:31.554601 7501 connection.cc:441] server connection from 127.0.0.1:45148 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45148 (request call id 1) could send its response
I20260929 14:42:31.560598 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.549284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.0.98.60:44347)
0929 14:42:31.549392 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.549397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.550468 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 14:42:31.550575 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.550578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.550627 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.550702 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.551824 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.552522 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.553837 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.554137 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.554236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.556007 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.556011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.556013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.556041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.558493 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.559029 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.559033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.559034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.559114 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.559378 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.559382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.559384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.559581 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.559703 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.560333 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.560445 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:42:31.561424 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.537835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.0.98.61:39265)
0929 14:42:31.537942 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.537946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.542232 (+ 4286us) server_negotiation.cc:407] Connection header received
0929 14:42:31.542289 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.542292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.542346 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.542431 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.550938 (+ 8507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.551756 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.553221 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.553565 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.553640 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.555981 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.555985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.555987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.556013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.557964 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.558423 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.558427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.558429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.558514 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.558834 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.558838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.558840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.559052 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.559172 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.559470 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.559579 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:42:31.563715 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.548666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.98.61:39265)
0929 14:42:31.548789 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.548793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.549839 (+ 1046us) server_negotiation.cc:407] Connection header received
0929 14:42:31.550035 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.550038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.550088 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.550144 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.555374 (+ 5230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.556048 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.557445 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.557804 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.557897 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.559615 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.559619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.559622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.559646 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.561261 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.562502 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.562505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.562507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.562584 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.562859 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.562863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.562864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.563063 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.563186 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.563468 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.563586 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:31.567420 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:31.537266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.0.98.60:44347)
0929 14:42:31.537368 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:31.537372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:31.560015 (+ 22643us) server_negotiation.cc:407] Connection header received
0929 14:42:31.560386 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:31.560389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:31.560434 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:31.560477 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:31.561188 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.561868 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.562692 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:31.562974 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:31.563070 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:31.564488 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:31.564491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:31.564494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:31.564517 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:31.565810 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.566234 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.566237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.566238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.566306 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:31.566530 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:31.566531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:31.566532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:31.566671 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:31.566753 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:31.567015 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 14:42:31.567266 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 14:42:32.519863 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:32.519360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:54199)
0929 14:42:32.519649 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:32.519746 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260929 14:42:32.520231 7614 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:32.521963 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:32.521539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:41449)
0929 14:42:32.521790 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:32.521865 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
I20260929 14:42:32.522286 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:32.521777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:59553)
0929 14:42:32.522092 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:32.522198 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":59,"threads_started":1}
W20260929 14:42:32.522315 7751 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
W20260929 14:42:32.522528 7477 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40905 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.035207 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e506138610264abd9f8ce3034ffddafc)
I20260929 14:42:33.035360 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } }
I20260929 14:42:33.035856 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905), 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60:44347)
I20260929 14:42:33.036515 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.036055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49300)
0929 14:42:33.036320 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.036410 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
W20260929 14:42:33.036768 7099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.037520 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.037331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49302)
0929 14:42:33.037426 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.037467 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:42:33.037643 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.038020 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e506138610264abd9f8ce3034ffddafc)
I20260929 14:42:33.038130 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } }
I20260929 14:42:33.038527 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905), 43ea92adb9fa4235b9277ad0ee5d961b (127.0.98.61:39265)
I20260929 14:42:33.039052 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.038665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49304)
0929 14:42:33.038877 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.038967 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
W20260929 14:42:33.039304 7282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.040179 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.039957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49320)
0929 14:42:33.040061 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.040118 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 14:42:33.040300 7282 leader_election.cc:334] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.040534 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.036412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44347 (local address 127.0.0.1:38162)
0929 14:42:33.036618 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.036632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.036741 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.037168 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.037171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.037189 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.037280 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.037285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.038235 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.038237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.039354 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.039362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.040150 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.040154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.040239 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.040262 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.040326 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.040399 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":55,"threads_started":1}
I20260929 14:42:33.040553 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.036633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.60:44347)
0929 14:42:33.036968 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.036972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.036982 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:42:33.037027 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.037030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.037068 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.037141 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.037381 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.038105 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.039471 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.040044 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.040135 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.040204 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.040341 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.040457 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260929 14:42:33.041087 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c3bdafc02074b0f949e37e84091d49f" is_pre_election: true
I20260929 14:42:33.041275 7303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ea92adb9fa4235b9277ad0ee5d961b in term 1.
I20260929 14:42:33.041586 7099 leader_election.cc:302] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43ea92adb9fa4235b9277ad0ee5d961b, 9c3bdafc02074b0f949e37e84091d49f; no voters: e506138610264abd9f8ce3034ffddafc
I20260929 14:42:33.041755 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:42:33.041819 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Starting leader election (detected failure of leader e506138610264abd9f8ce3034ffddafc)
I20260929 14:42:33.041858 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:42:33.042640 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.038942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39265 (local address 127.0.0.1:47488)
0929 14:42:33.039153 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.039164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.039236 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.039576 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.039579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.039598 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.039678 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.039683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.040546 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.040550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.041563 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.041573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.042340 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.042343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.042407 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.042427 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.042485 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.042533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":54,"threads_started":1}
I20260929 14:42:33.042677 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.039025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.0.98.61:39265)
0929 14:42:33.039362 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.039372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.039386 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:33.039426 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.039428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.039470 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.039560 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.039775 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.040445 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.041677 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.042237 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.042334 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.042386 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.042514 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.042575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":95,"threads_started":1}
I20260929 14:42:33.043094 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } }
I20260929 14:42:33.043203 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c3bdafc02074b0f949e37e84091d49f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" is_pre_election: true
I20260929 14:42:33.043360 7124 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9c3bdafc02074b0f949e37e84091d49f in current term 2: Already voted for candidate 43ea92adb9fa4235b9277ad0ee5d961b in this term.
I20260929 14:42:33.043401 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 election: Requested vote from peers e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905), 9c3bdafc02074b0f949e37e84091d49f (127.0.98.60:44347)
I20260929 14:42:33.043663 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.043497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49324)
0929 14:42:33.043574 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.043614 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 14:42:33.043746 7282 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c3bdafc02074b0f949e37e84091d49f; no voters: 43ea92adb9fa4235b9277ad0ee5d961b, e506138610264abd9f8ce3034ffddafc
I20260929 14:42:33.043983 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c3bdafc02074b0f949e37e84091d49f"
I20260929 14:42:33.044080 7303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:42:33.044270 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.044081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49328)
0929 14:42:33.044169 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.044215 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 14:42:33.044346 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
I20260929 14:42:33.045099 7303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ea92adb9fa4235b9277ad0ee5d961b in term 2.
I20260929 14:42:33.045208 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:42:33.045352 7099 leader_election.cc:302] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43ea92adb9fa4235b9277ad0ee5d961b, 9c3bdafc02074b0f949e37e84091d49f; no voters: e506138610264abd9f8ce3034ffddafc
I20260929 14:42:33.045593 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:42:33.045799 7820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [term 2 LEADER]: Becoming Leader. State: Replica: 43ea92adb9fa4235b9277ad0ee5d961b, State: Running, Role: LEADER
I20260929 14:42:33.046164 7820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } }
I20260929 14:42:33.047215 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ea92adb9fa4235b9277ad0ee5d961b. Latest consensus state: current_term: 2 leader_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:33.047343 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:33.047639 7832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:33.049889 7832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:33.050272 7832 catalog_manager.cc:1259] Loaded cluster ID: 62c6a16bba614ba7a14d79c04bba8b0f
I20260929 14:42:33.050318 7832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:33.050848 7832 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:33.051208 7832 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Loaded TSK: 0
I20260929 14:42:33.051883 7832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:33.090911 7107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:33.094338 7107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:33.097383 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.097096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49354)
0929 14:42:33.097255 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.097312 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 14:42:33.097646 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [term 2 FOLLOWER]: Refusing update from remote peer 43ea92adb9fa4235b9277ad0ee5d961b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 14:42:33.098086 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.097882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.0.1:49368)
0929 14:42:33.097973 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.098019 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 14:42:33.098158 7820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 14:42:33.098327 7099 consensus_peers.cc:487] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b -> Peer e506138610264abd9f8ce3034ffddafc (127.0.98.62:40905): Couldn't send request to peer e506138610264abd9f8ce3034ffddafc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:42:33.099668 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43ea92adb9fa4235b9277ad0ee5d961b. Latest consensus state: current_term: 2 leader_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:33.099802 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:33.100934 7830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:33.101042 7830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:33.101475 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e506138610264abd9f8ce3034ffddafc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40905 } } peers { permanent_uuid: "43ea92adb9fa4235b9277ad0ee5d961b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39265 } attrs { promote: false } } peers { permanent_uuid: "9c3bdafc02074b0f949e37e84091d49f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44347 } attrs { promote: false } } }
I20260929 14:42:33.101608 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3bdafc02074b0f949e37e84091d49f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:42:33.101483 7834 mvcc.cc:205] Tried to move back new op lower bound from 7334678335882452992 to 7334678335679500288. Current Snapshot: MvccSnapshot[applied={T|T < 7334678335882452992}]
I20260929 14:42:33.111093 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.106588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39919 (local address 127.0.0.1:43478)
0929 14:42:33.106728 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.106741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.106804 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.107237 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.107240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.107256 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.107358 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.107365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.108353 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.108356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.109525 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.109532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.110716 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.110801 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.110824 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.110890 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.110960 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 14:42:33.111202 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.107059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.0.1:57184)
0929 14:42:33.107224 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.107237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.107312 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.108122 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.108125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.108140 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.108234 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.108240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.109091 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.109094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.109964 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.109972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.110838 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.110919 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.110937 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.110995 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.111051 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":2,"spinlock_wait_cycles":22016}
I20260929 14:42:33.111225 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.107688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.0.98.1:44371)
0929 14:42:33.107889 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.107894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.107908 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:33.107972 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.107975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.108023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.108102 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.108335 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.108994 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110091 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.110721 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110843 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.110906 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.111016 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.111094 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":56,"threads_started":1}
I20260929 14:42:33.111613 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.106719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.98.2:39919)
0929 14:42:33.107001 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.107006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.107021 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:33.107068 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.107071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.107121 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.107467 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.107575 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.108281 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.109640 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.110605 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110703 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.110768 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.111395 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.111477 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260929 14:42:33.111752 7411 tablet_service.cc:1467] Processing CreateTablet for tablet cc71149a009941d4a912d147aacc4dc4 (DEFAULT_TABLE table=test-table [id=e7749aaa6e8d4e44936d9ee58f1c5d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:33.112062 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc71149a009941d4a912d147aacc4dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:33.112028 7548 tablet_service.cc:1467] Processing CreateTablet for tablet cc71149a009941d4a912d147aacc4dc4 (DEFAULT_TABLE table=test-table [id=e7749aaa6e8d4e44936d9ee58f1c5d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:33.112231 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc71149a009941d4a912d147aacc4dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:33.112268 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.107447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.98.3:33323)
0929 14:42:33.107699 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.107703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.108111 (+ 408us) server_negotiation.cc:407] Connection header received
0929 14:42:33.108270 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.108274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.108323 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.108400 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.108613 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.109401 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.110827 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.111385 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.111448 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.111500 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.112076 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.112154 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
I20260929 14:42:33.112658 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.107067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33323 (local address 127.0.0.1:50218)
0929 14:42:33.108034 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.108047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.108178 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.108406 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.108409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.108426 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.108501 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.108507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.109533 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.109536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.110687 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.110697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.111801 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.111804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.111925 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.111947 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.112454 (+ 507us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.112537 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":746,"thread_start_us":81,"threads_started":1}
I20260929 14:42:33.114449 7843 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Bootstrap starting.
I20260929 14:42:33.114485 7844 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: Bootstrap starting.
I20260929 14:42:33.114876 7685 tablet_service.cc:1467] Processing CreateTablet for tablet cc71149a009941d4a912d147aacc4dc4 (DEFAULT_TABLE table=test-table [id=e7749aaa6e8d4e44936d9ee58f1c5d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:33.115159 7685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc71149a009941d4a912d147aacc4dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:33.115356 7844 tablet_bootstrap.cc:654] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:33.115366 7843 tablet_bootstrap.cc:654] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:33.116444 7843 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: No bootstrap required, opened a new log
I20260929 14:42:33.116528 7843 ts_tablet_manager.cc:1397] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:33.116539 7844 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: No bootstrap required, opened a new log
I20260929 14:42:33.116617 7844 ts_tablet_manager.cc:1397] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:33.116997 7843 raft_consensus.cc:348] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.117115 7843 raft_consensus.cc:374] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:33.117147 7843 raft_consensus.cc:729] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0828b6e790f746eca979c0d54b186fed, State: Initialized, Role: FOLLOWER
I20260929 14:42:33.117103 7844 raft_consensus.cc:348] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.117208 7844 raft_consensus.cc:374] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:33.117238 7844 raft_consensus.cc:729] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91aae8b7928041d9876cfc190512f8e8, State: Initialized, Role: FOLLOWER
I20260929 14:42:33.117358 7844 consensus_queue.cc:260] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.117303 7843 consensus_queue.cc:260] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.117657 7844 ts_tablet_manager.cc:1428] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:33.117661 7843 ts_tablet_manager.cc:1428] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:33.118619 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.118206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:34473)
0929 14:42:33.118454 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.118515 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260929 14:42:33.119570 7847 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Bootstrap starting.
I20260929 14:42:33.119867 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.118378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:56161)
0929 14:42:33.119712 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.119774 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":82,"threads_started":1}
I20260929 14:42:33.120585 7847 tablet_bootstrap.cc:654] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:33.120802 7616 heartbeater.cc:503] Master 127.0.98.61:39265 was elected leader, sending a full tablet report...
I20260929 14:42:33.121541 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.120320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:41831)
0929 14:42:33.121418 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.121469 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1047,"thread_start_us":82,"threads_started":1}
I20260929 14:42:33.121909 7847 tablet_bootstrap.cc:492] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: No bootstrap required, opened a new log
I20260929 14:42:33.121987 7847 ts_tablet_manager.cc:1397] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:42:33.122357 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.121445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:57825)
0929 14:42:33.122235 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.122289 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720}
I20260929 14:42:33.122401 7847 raft_consensus.cc:348] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.122515 7847 raft_consensus.cc:374] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:33.122555 7847 raft_consensus.cc:729] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cbe9e1104fb4153ad340681d9da5760, State: Initialized, Role: FOLLOWER
I20260929 14:42:33.122699 7847 consensus_queue.cc:260] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.123013 7847 ts_tablet_manager.cc:1428] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:33.123602 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.123256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:42557)
0929 14:42:33.123456 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.123517 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
I20260929 14:42:33.124368 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.124159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:52169)
0929 14:42:33.124255 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.124306 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 14:42:33.124396 7479 heartbeater.cc:503] Master 127.0.98.61:39265 was elected leader, sending a full tablet report...
I20260929 14:42:33.124392 7753 heartbeater.cc:503] Master 127.0.98.61:39265 was elected leader, sending a full tablet report...
I20260929 14:42:33.321192 7848 raft_consensus.cc:482] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:33.321355 7848 raft_consensus.cc:504] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.322546 7848 leader_election.cc:288] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919), 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
I20260929 14:42:33.326499 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.322914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40291 (local address 127.0.98.2:39919)
0929 14:42:33.323041 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.323047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.323304 (+ 257us) server_negotiation.cc:407] Connection header received
0929 14:42:33.323432 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.323435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.323479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.323532 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.323907 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.324520 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.325618 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.326019 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.326119 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.326189 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.326306 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.326374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:33.326506 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.322826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39919 (local address 127.0.98.1:40291)
0929 14:42:33.323201 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.323219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.323340 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.323568 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.323571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.323587 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.323778 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.323792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.324623 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.324626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.325500 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.325511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.326129 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.326132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.326204 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.326221 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.326286 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.326352 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":47,"spinlock_wait_cycles":17152,"thread_start_us":146,"threads_started":1}
I20260929 14:42:33.326499 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.322904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56167 (local address 127.0.98.3:33323)
0929 14:42:33.323045 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.323050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.323301 (+ 251us) server_negotiation.cc:407] Connection header received
0929 14:42:33.323439 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.323443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.323501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.323577 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:33.323854 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.324447 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.325529 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.325938 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.326014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.326069 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.326287 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.326372 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:33.326510 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.322823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33323 (local address 127.0.98.1:56167)
0929 14:42:33.323200 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.323212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:33.323336 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:33.323605 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:33.323608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:33.323623 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:33.323773 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.323778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.324543 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.324546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.325414 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:33.325423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.326067 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.326071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:33.326169 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.326191 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:42:33.326259 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:42:33.326328 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":31,"spinlock_wait_cycles":6016}
I20260929 14:42:33.326948 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc71149a009941d4a912d147aacc4dc4" candidate_uuid: "91aae8b7928041d9876cfc190512f8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0828b6e790f746eca979c0d54b186fed" is_pre_election: true
I20260929 14:42:33.327111 7568 raft_consensus.cc:2454] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91aae8b7928041d9876cfc190512f8e8 in term 0.
I20260929 14:42:33.327165 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc71149a009941d4a912d147aacc4dc4" candidate_uuid: "91aae8b7928041d9876cfc190512f8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbe9e1104fb4153ad340681d9da5760" is_pre_election: true
I20260929 14:42:33.327389 7705 raft_consensus.cc:2454] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91aae8b7928041d9876cfc190512f8e8 in term 0.
I20260929 14:42:33.327491 7364 leader_election.cc:302] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0828b6e790f746eca979c0d54b186fed, 91aae8b7928041d9876cfc190512f8e8; no voters:
I20260929 14:42:33.327780 7848 raft_consensus.cc:2781] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:33.327836 7848 raft_consensus.cc:482] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:33.327862 7848 raft_consensus.cc:3037] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:33.328960 7848 raft_consensus.cc:504] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.329298 7848 leader_election.cc:288] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [CANDIDATE]: Term 1 election: Requested vote from peers 0828b6e790f746eca979c0d54b186fed (127.0.98.2:39919), 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323)
I20260929 14:42:33.329618 7705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc71149a009941d4a912d147aacc4dc4" candidate_uuid: "91aae8b7928041d9876cfc190512f8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cbe9e1104fb4153ad340681d9da5760"
I20260929 14:42:33.329618 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc71149a009941d4a912d147aacc4dc4" candidate_uuid: "91aae8b7928041d9876cfc190512f8e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0828b6e790f746eca979c0d54b186fed"
I20260929 14:42:33.329739 7568 raft_consensus.cc:3037] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:33.329739 7705 raft_consensus.cc:3037] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:33.330811 7568 raft_consensus.cc:2454] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91aae8b7928041d9876cfc190512f8e8 in term 1.
I20260929 14:42:33.330811 7705 raft_consensus.cc:2454] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91aae8b7928041d9876cfc190512f8e8 in term 1.
I20260929 14:42:33.331072 7364 leader_election.cc:302] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0828b6e790f746eca979c0d54b186fed, 91aae8b7928041d9876cfc190512f8e8; no voters:
I20260929 14:42:33.331322 7848 raft_consensus.cc:2781] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:33.331525 7848 raft_consensus.cc:686] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [term 1 LEADER]: Becoming Leader. State: Replica: 91aae8b7928041d9876cfc190512f8e8, State: Running, Role: LEADER
I20260929 14:42:33.331868 7848 consensus_queue.cc:237] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } }
I20260929 14:42:33.332212 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.331966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:42285)
0929 14:42:33.332071 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.332122 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 14:42:33.334165 7115 catalog_manager.cc:5495] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "91aae8b7928041d9876cfc190512f8e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91aae8b7928041d9876cfc190512f8e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44371 } health_report { overall_health: HEALTHY } } }
I20260929 14:42:33.347316 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.343337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.0.98.1:44371)
0929 14:42:33.343478 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:33.343482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:33.343494 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:33.343566 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:33.343569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:33.343625 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:33.343714 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:33.343985 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.344527 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.345808 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:33.346040 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:33.346133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:33.346444 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:33.346916 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:33.347005 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:33.347130 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 14:42:33.347184 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:33.349990 7568 raft_consensus.cc:1261] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Refusing update from remote peer 91aae8b7928041d9876cfc190512f8e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:33.350150 7705 raft_consensus.cc:1261] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Refusing update from remote peer 91aae8b7928041d9876cfc190512f8e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:33.350596 7848 consensus_queue.cc:1035] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0828b6e790f746eca979c0d54b186fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:33.350944 7848 consensus_queue.cc:1035] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cbe9e1104fb4153ad340681d9da5760" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:33.352330 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.352066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:57935)
0929 14:42:33.352162 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.352227 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 14:42:33.354656 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.353601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:38759)
0929 14:42:33.354546 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.354596 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":881}
I20260929 14:42:33.355935 7863 mvcc.cc:205] Tried to move back new op lower bound from 7334678336916623360 to 7334678336849895424. Current Snapshot: MvccSnapshot[applied={T|T < 7334678336916623360}]
I20260929 14:42:33.356449 7861 mvcc.cc:205] Tried to move back new op lower bound from 7334678336916623360 to 7334678336849895424. Current Snapshot: MvccSnapshot[applied={T|T < 7334678336916623360}]
I20260929 14:42:33.356684 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.356336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.3:38223)
0929 14:42:33.356567 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.356618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260929 14:42:33.357381 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.357020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.2:60493)
0929 14:42:33.357263 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.357313 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260929 14:42:33.358145 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.357717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:35125)
0929 14:42:33.358035 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.358085 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258}
I20260929 14:42:33.360268 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.359713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40905 (local address 127.0.98.1:49647)
0929 14:42:33.360154 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.360203 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
I20260929 14:42:33.365130 7114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:33.365264 7114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:33.368785 7114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43ea92adb9fa4235b9277ad0ee5d961b: Sending DeleteTablet for 3 replicas of tablet cc71149a009941d4a912d147aacc4dc4
I20260929 14:42:33.369717 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet cc71149a009941d4a912d147aacc4dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:33 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43478
I20260929 14:42:33.369706 7685 tablet_service.cc:1514] Processing DeleteTablet for tablet cc71149a009941d4a912d147aacc4dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:33 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50218
I20260929 14:42:33.370035 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet cc71149a009941d4a912d147aacc4dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:33 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:57184
I20260929 14:42:33.370342 7872 tablet_replica.cc:330] T cc71149a009941d4a912d147aacc4dc4 P 91aae8b7928041d9876cfc190512f8e8: stopping tablet replica
I20260929 14:42:33.370580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7260
I20260929 14:42:33.371343 7871 tablet_replica.cc:330] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: stopping tablet replica
I20260929 14:42:33.371457 7871 raft_consensus.cc:2229] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:33.371596 7871 raft_consensus.cc:2258] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:33.371906 7870 tablet_replica.cc:330] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: stopping tablet replica
I20260929 14:42:33.371995 7870 raft_consensus.cc:2229] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:33.372089 7870 raft_consensus.cc:2258] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:33.372797 7871 ts_tablet_manager.cc:1905] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:33.372937 7870 ts_tablet_manager.cc:1905] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:33.375391 7871 ts_tablet_manager.cc:1918] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:33.375427 7870 ts_tablet_manager.cc:1918] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:33.375471 7871 log.cc:1196] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/cc71149a009941d4a912d147aacc4dc4
I20260929 14:42:33.375488 7870 log.cc:1196] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/cc71149a009941d4a912d147aacc4dc4
I20260929 14:42:33.375752 7870 ts_tablet_manager.cc:1939] T cc71149a009941d4a912d147aacc4dc4 P 0cbe9e1104fb4153ad340681d9da5760: Deleting consensus metadata
I20260929 14:42:33.375752 7871 ts_tablet_manager.cc:1939] T cc71149a009941d4a912d147aacc4dc4 P 0828b6e790f746eca979c0d54b186fed: Deleting consensus metadata
I20260929 14:42:33.376485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7485
I20260929 14:42:33.376786 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.376500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44371 (local address 127.0.0.1:57200)
0929 14:42:33.376642 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.376700 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 14:42:33.378054 7101 catalog_manager.cc:4841] TS 0cbe9e1104fb4153ad340681d9da5760 (127.0.98.3:33323): tablet cc71149a009941d4a912d147aacc4dc4 (table test-table [id=e7749aaa6e8d4e44936d9ee58f1c5d60]) successfully deleted
W20260929 14:42:33.378417 7099 catalog_manager.cc:4584] TS 91aae8b7928041d9876cfc190512f8e8 (127.0.98.1:44371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc71149a009941d4a912d147aacc4dc4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44371: connect: Connection refused (error 111)
I20260929 14:42:33.381938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7621
I20260929 14:42:33.382128 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:33.381862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39919 (local address 127.0.0.1:43488)
0929 14:42:33.381986 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:33.382040 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 14:42:33.388388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 6977
I20260929 14:42:33.393630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7081
2026-09-29T14:42:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5482 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:33 dist-test-slave-g8hp krb5kdc[7875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:33 dist-test-slave-g8hp krb5kdc[7875](info): set up 2 sockets
Sep 29 14:42:33 dist-test-slave-g8hp krb5kdc[7875](info): commencing operation
krb5kdc: starting...
W20260929 14:42:34.353328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:34 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 14:42:34.473907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:34.583441 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:34.583725 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:34.583765 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:34.587898 7888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:34.587981 7888 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 14:42:34.587997 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:34.588014 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:34.588029 7888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:34.588043 7888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:34.593690 7888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41621
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:34.594821 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:34.595887 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:34.604974 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:34.605041 7888 server_base.cc:1034] running on GCE node
W20260929 14:42:34.605244 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:34.604981 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:34.605894 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:34.607077 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:34.608269 7888 hybrid_clock.cc:648] HybridClock initialized: now 1790692954608240 us; error 47 us; skew 500 ppm
Sep 29 14:42:34 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:34.611138 7888 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 14:42:34.612097 7888 webserver.cc:460] Webserver started at http://127.0.98.62:38027/ using document root <none> and password file <none>
I20260929 14:42:34.612632 7888 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:34.612694 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:34.612905 7888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:34.615028 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "2baa94cc5a41cda55208f7422215391d"
server_key_iv: "4ea311b321c5adc50315d34108ae6fd0"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.615566 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "2baa94cc5a41cda55208f7422215391d"
server_key_iv: "4ea311b321c5adc50315d34108ae6fd0"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.619336 7888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:42:34.621948 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:34.623306 7888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:34.623421 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "2baa94cc5a41cda55208f7422215391d"
server_key_iv: "4ea311b321c5adc50315d34108ae6fd0"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.623553 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:34.645690 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:34.648898 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:34.649094 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:34.656929 7888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41621
I20260929 14:42:34.657006 7965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41621 every 8 connection(s)
I20260929 14:42:34.657938 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:34.660382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7888
I20260929 14:42:34.660480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:34.660763 392 external_mini_cluster.cc:1428] Setting key 0180bee6706be78f7822dd68083f1337
I20260929 14:42:34.660935 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:34.667960 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Bootstrap starting.
Sep 29 14:42:34 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:34.670775 7966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:34.671619 7966 log.cc:824] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:34.674046 7966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: No bootstrap required, opened a new log
I20260929 14:42:34.677076 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:34.662218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.98.62:41621)
0929 14:42:34.662624 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:34.662639 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:34.662675 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:34.663641 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:34.663659 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:34.664023 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:34.664311 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:34.665293 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.666338 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.667516 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.668154 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.668310 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:34.671191 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:34.671198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:34.671216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:34.671250 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:34.673846 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.674314 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.674320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.674326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.674477 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.674704 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.674707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.674708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.675228 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:34.675424 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:34.675728 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 14:42:34.676046 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260929 14:42:34.677892 7966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260929 14:42:34.678113 7966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:34.678148 7966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bdf1a1f96ae4fb2a92e5de5d77f10de, State: Initialized, Role: FOLLOWER
I20260929 14:42:34.678583 7966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260929 14:42:34.678710 7966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:34.678773 7966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:34.678867 7966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:34.679774 7966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260929 14:42:34.680097 7966 leader_election.cc:302] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bdf1a1f96ae4fb2a92e5de5d77f10de; no voters:
I20260929 14:42:34.680380 7966 leader_election.cc:288] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:34.680536 7971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:34.680737 7971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [term 1 LEADER]: Becoming Leader. State: Replica: 6bdf1a1f96ae4fb2a92e5de5d77f10de, State: Running, Role: LEADER
I20260929 14:42:34.681071 7971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } }
I20260929 14:42:34.681690 7966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:34.682643 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } }
I20260929 14:42:34.682767 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:34.683130 7980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:34.683101 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bdf1a1f96ae4fb2a92e5de5d77f10de. Latest consensus state: current_term: 1 leader_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bdf1a1f96ae4fb2a92e5de5d77f10de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41621 } } }
I20260929 14:42:34.683744 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:34.686370 7980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:34.692253 7980 catalog_manager.cc:1347] Generated new cluster ID: 2bc8f35ae4604621977d8dffbd86768c
I20260929 14:42:34.692337 7980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:34.717559 7980 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:34.718683 7980 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:34.725309 7980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Generated new TSK 0
I20260929 14:42:34.726112 7980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 14:42:34.762006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41621
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:34.875835 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:34.876135 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:34.876199 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:34.880425 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:34.880509 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:34.880568 7992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:34.880620 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:34.886343 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:34.887624 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:34.888720 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:34.898679 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:34.898665 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:34.898665 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:34.898747 7992 server_base.cc:1034] running on GCE node
I20260929 14:42:34.899415 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:34.900110 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:34.901335 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790692954901302 us; error 48 us; skew 500 ppm
Sep 29 14:42:34 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:34.904342 7992 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:42:34.905295 7992 webserver.cc:460] Webserver started at http://127.0.98.1:46081/ using document root <none> and password file <none>
I20260929 14:42:34.905850 7992 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:34.905915 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:34.906121 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:34.908243 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "e7c3383dc73d4c35b786f53b97435604"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "7f71eac18101726e1134dcd369d7d4ed"
server_key_iv: "7749476522c573da874a3b3da858f296"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.908778 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c3383dc73d4c35b786f53b97435604"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "7f71eac18101726e1134dcd369d7d4ed"
server_key_iv: "7749476522c573da874a3b3da858f296"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.912531 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:34.915132 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:34.916474 7992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:34.916582 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "e7c3383dc73d4c35b786f53b97435604"
format_stamp: "Formatted at 2026-09-29 14:42:34 on dist-test-slave-g8hp"
server_key: "7f71eac18101726e1134dcd369d7d4ed"
server_key_iv: "7749476522c573da874a3b3da858f296"
server_key_version: "encryptionkey@0"
I20260929 14:42:34.916689 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:34.928929 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:34.932257 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:34.932413 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:34.933876 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:34.933923 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:34.933962 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:34.933975 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:34 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:34.944185 7992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32931
I20260929 14:42:34.944216 8122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32931 every 8 connection(s)
I20260929 14:42:34.945365 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:34.947553 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:34.934824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49499 (local address 127.0.98.62:41621)
0929 14:42:34.935011 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:34.935014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:34.935940 (+ 926us) server_negotiation.cc:407] Connection header received
0929 14:42:34.937116 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:34.937120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:34.937176 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:34.937263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:34.938835 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.939404 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.940616 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.940846 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.940974 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:34.943681 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:34.943685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:34.943688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:34.943720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:34.945396 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.946018 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.946022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.946024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.946122 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.946483 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.946487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.946489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.946706 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:34.946792 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:34.947278 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 14:42:34.947428 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 14:42:34.948329 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:34.935300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.1:49499)
0929 14:42:34.935770 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:34.935813 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:34.936817 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:34.937483 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:34.937507 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:34.937965 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:34.938666 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:34.938687 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.939534 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.939537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:34.940499 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:34.940509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.941032 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.941036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:34.941193 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:34.941833 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:34.941851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:34.943488 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:34.945537 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:34.945543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:34.945555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:34.945881 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:34.946218 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:34.946220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:34.946222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:34.946363 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:34.946808 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:34.946813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:34.947062 (+ 249us) client_negotiation.cc:769] Sending connection context
0929 14:42:34.947219 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 14:42:34.947475 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260929 14:42:34.948704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 7992
I20260929 14:42:34.948814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:34.949172 392 external_mini_cluster.cc:1428] Setting key 555bc0ebab2b58443b1ef6f943fdfec7
I20260929 14:42:34.949728 8123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41621
I20260929 14:42:34.949875 8123 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:34.950416 8123 heartbeater.cc:511] Master 127.0.98.62:41621 requested a full tablet report, sending...
I20260929 14:42:34.952102 7919 ts_manager.cc:194] Registered new tserver with Master: e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931)
I20260929 14:42:34.953517 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49499
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:34.959903 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:34.951974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45199 (local address 127.0.98.62:41621)
0929 14:42:34.952080 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:34.952084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:34.952338 (+ 254us) server_negotiation.cc:407] Connection header received
0929 14:42:34.952451 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:34.952454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:34.952502 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:34.952580 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:34.953123 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.953836 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.955011 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.955347 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.955456 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:34.956742 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:34.956746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:34.956749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:34.956774 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:34.958330 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.958737 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.958741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.958743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.958815 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:34.959098 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:34.959102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:34.959104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:34.959295 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:34.959383 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:34.959637 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 14:42:34.959748 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:34.959930 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:34.951893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.1:45199)
0929 14:42:34.952242 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:34.952257 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:34.952374 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:34.952618 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:34.952621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:34.952851 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:34.953006 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:34.953014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.953998 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.954002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:34.954867 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:34.954879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:34.955477 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:34.955481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:34.955575 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:34.956197 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:34.956209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:34.956590 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:34.958444 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:34.958448 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:34.958450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:34.958646 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:34.958897 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:34.958900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:34.958902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:34.959014 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:34.959388 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:34.959391 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:34.959485 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 14:42:34.959571 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 14:42:34.959732 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 14:42:34.981447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41621
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:35.095492 8132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:35.095762 8132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:35.095808 8132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:35.100057 8132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:35.100129 8132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:35.100170 8132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:35.100203 8132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:35.105666 8132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:35.106747 8132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:35.107812 8132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:35.117759 8140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:35.117925 8132 server_base.cc:1034] running on GCE node
W20260929 14:42:35.117865 8138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:35.117789 8137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:35.118533 8132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:35.119131 8132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:35.120302 8132 hybrid_clock.cc:648] HybridClock initialized: now 1790692955120269 us; error 42 us; skew 500 ppm
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:35.123121 8132 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:42:35.124035 8132 webserver.cc:460] Webserver started at http://127.0.98.2:41357/ using document root <none> and password file <none>
I20260929 14:42:35.124557 8132 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:35.124615 8132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:35.124831 8132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:35.127023 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "4b95b20b9c79b4c8bdec501c9ed2fbfc"
server_key_iv: "5f884121ecedb62fb1603ee25cc00e7a"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.127612 8132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "4b95b20b9c79b4c8bdec501c9ed2fbfc"
server_key_iv: "5f884121ecedb62fb1603ee25cc00e7a"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.131343 8132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260929 14:42:35.133855 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:35.135041 8132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:35.135146 8132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "4b95b20b9c79b4c8bdec501c9ed2fbfc"
server_key_iv: "5f884121ecedb62fb1603ee25cc00e7a"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.135237 8132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:35.161767 8132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:35.165050 8132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:35.165230 8132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:35.166677 8132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:35.166728 8132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:35.166780 8132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:35.166810 8132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:35.176607 8132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34695
I20260929 14:42:35.176685 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34695 every 8 connection(s)
I20260929 14:42:35.177544 8132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:35.178544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8132
I20260929 14:42:35.178666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:35.178948 392 external_mini_cluster.cc:1428] Setting key 61bf9821b6539ee297c67a36b4f8d1d6
I20260929 14:42:35.181032 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.167661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50651 (local address 127.0.98.62:41621)
0929 14:42:35.167808 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.167813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.168727 (+ 914us) server_negotiation.cc:407] Connection header received
0929 14:42:35.170103 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.170107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.170171 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.170226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.172074 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.172805 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.174116 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.174339 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.174432 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.177116 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.177120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.177122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.177148 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.178688 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.179164 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.179168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.179170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.179270 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.179646 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.179649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.179651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.179867 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.179949 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.180566 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.180773 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:42:35.181602 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.168076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.2:50651)
0929 14:42:35.168562 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.168627 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.169802 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.170639 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.170649 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.171168 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:35.171888 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.171911 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.172980 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.172983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.173988 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.173998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.174491 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.174495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.174692 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.175308 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:35.175324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:35.176939 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:35.178788 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.178793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.178808 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.179072 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.179385 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.179388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.179389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.179576 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.179981 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:35.179987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:35.180287 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.180481 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.180757 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260929 14:42:35.183216 8263 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41621
I20260929 14:42:35.183375 8263 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:35.183928 8263 heartbeater.cc:511] Master 127.0.98.62:41621 requested a full tablet report, sending...
I20260929 14:42:35.184942 7919 ts_manager.cc:194] Registered new tserver with Master: fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695)
I20260929 14:42:35.185664 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50651
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:35.192584 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.185486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.2:35099)
0929 14:42:35.185863 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.185879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.185969 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.186256 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.186258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.186438 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:35.186529 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.186533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.187514 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.187517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.188094 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.188102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.188539 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.188542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.188634 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.189169 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:35.189181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:35.189481 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:35.191117 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.191120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.191122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.191304 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.191558 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.191561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.191564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.191642 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.192088 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:35.192091 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:35.192177 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.192319 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.192440 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260929 14:42:35.192728 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.185784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35099 (local address 127.0.98.62:41621)
0929 14:42:35.185913 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.185917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.185932 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:35.186072 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.186075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.186117 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.186154 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.186632 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.187370 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.188202 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.188435 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.188522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.189604 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.189608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.189610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.189634 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.191020 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.191386 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.191389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.191391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.191470 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.191774 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.191777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.191779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.191998 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.192086 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.192404 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.192573 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:42:35.208614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41621
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:35.317920 8272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:35.318162 8272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:35.318222 8272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:35.322352 8272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:35.322418 8272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:35.322474 8272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 14:42:35.322535 8272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:35.328019 8272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:35.329149 8272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:35.330145 8272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:35.341601 8277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:35.341763 8272 server_base.cc:1034] running on GCE node
W20260929 14:42:35.341601 8280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:35.341615 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:35.342301 8272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:35.343084 8272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:35.344271 8272 hybrid_clock.cc:648] HybridClock initialized: now 1790692955344198 us; error 89 us; skew 500 ppm
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:35.347597 8272 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:42:35.348489 8272 webserver.cc:460] Webserver started at http://127.0.98.3:40823/ using document root <none> and password file <none>
I20260929 14:42:35.349339 8272 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:35.349406 8272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:35.349644 8272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:35.351748 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "eed413db696e4ee69138448595f5ac64"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "f5ff0f92892d7ddeaaa8cc68718c649e"
server_key_iv: "0ff52919c4b10a4e1c584c613e2cf91d"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.352288 8272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eed413db696e4ee69138448595f5ac64"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "f5ff0f92892d7ddeaaa8cc68718c649e"
server_key_iv: "0ff52919c4b10a4e1c584c613e2cf91d"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.356345 8272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 14:42:35.358937 8287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:35.360345 8272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:35.360491 8272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "eed413db696e4ee69138448595f5ac64"
format_stamp: "Formatted at 2026-09-29 14:42:35 on dist-test-slave-g8hp"
server_key: "f5ff0f92892d7ddeaaa8cc68718c649e"
server_key_iv: "0ff52919c4b10a4e1c584c613e2cf91d"
server_key_version: "encryptionkey@0"
I20260929 14:42:35.360591 8272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:35.370270 8272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:35.373404 8272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:35.373576 8272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:35.374949 8272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:35.375000 8272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:35.375052 8272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:35.375077 8272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:35.385157 8272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33987
I20260929 14:42:35.385217 8402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33987 every 8 connection(s)
I20260929 14:42:35.386040 8272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:35.388530 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.375976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37981 (local address 127.0.98.62:41621)
0929 14:42:35.376098 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.376103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.377196 (+ 1093us) server_negotiation.cc:407] Connection header received
0929 14:42:35.378383 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.378387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.378451 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.378549 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.379984 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.380791 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.382126 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.382427 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.382527 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.385333 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.385337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.385340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.385369 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.386762 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.387288 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.387290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.387292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.387367 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.387658 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.387660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.387661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.387825 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.387906 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.388314 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.388411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 14:42:35.389436 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.376500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.3:37981)
0929 14:42:35.377046 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.377092 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.378130 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.378712 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.378719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.379152 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:35.379806 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.379826 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.380962 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.380965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.382016 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.382028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.382578 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.382582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.382758 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.383385 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:35.383405 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:35.385125 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:35.386888 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.386893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.386906 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.387168 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.387470 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.387472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.387474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.387595 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.387908 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:35.387912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:35.388130 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.388274 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.388521 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260929 14:42:35.390311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8272
I20260929 14:42:35.390426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:35.390686 392 external_mini_cluster.cc:1428] Setting key dfd525b8a30757f48082e6425ba64eb4
I20260929 14:42:35.391069 8403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41621
I20260929 14:42:35.391225 8403 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:35.391985 8403 heartbeater.cc:511] Master 127.0.98.62:41621 requested a full tablet report, sending...
I20260929 14:42:35.393049 7920 ts_manager.cc:194] Registered new tserver with Master: eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:35.393697 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37981
I20260929 14:42:35.399811 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.392701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.98.3:34747)
0929 14:42:35.392959 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.392972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.393058 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.393256 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.393258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.393414 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:35.393506 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.393511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.394383 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.394387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.395194 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.395204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.395697 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.395700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.395795 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.396254 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:35.396264 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:35.396591 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:35.398534 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.398537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.398538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.398720 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.398996 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.398999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.399001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.399070 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.399378 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:35.399380 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:35.399456 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.399553 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.399661 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260929 14:42:35.399859 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.392681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34747 (local address 127.0.98.62:41621)
0929 14:42:35.392795 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.392799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.393045 (+ 246us) server_negotiation.cc:407] Connection header received
0929 14:42:35.393119 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.393122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.393166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.393237 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.393597 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.394265 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.395350 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.395576 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.395669 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.396710 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.396713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.396715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.396735 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.398430 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.398792 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.398795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.398796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.398864 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.399141 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.399143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.399145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.399313 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.399383 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.399648 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.399745 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":40}
I20260929 14:42:35.404342 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:42:35.426831 7969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:35.420690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.0.98.62:41621)
0929 14:42:35.420831 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.420834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.420846 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:35.421037 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.421041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.421085 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.421157 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.421899 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.422428 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.423247 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.423474 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.423562 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.426666 (+ 3104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43434: BlockingRecv error: recv got EOF from 127.0.0.1:43434 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:35.435712 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.428102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.98.62:41621)
0929 14:42:35.428196 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.428200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.428330 (+ 130us) server_negotiation.cc:407] Connection header received
0929 14:42:35.428489 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.428491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.428530 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.428608 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.429332 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.429828 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.430598 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.430818 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.430902 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.433083 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.433086 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.433088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.433111 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.434194 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.434612 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.434615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.434616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.434687 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.434928 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.434931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.434933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.435107 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.435229 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.435444 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.435551 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:42:35.438606 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:35.441680 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:35.455719 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.449781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34695 (local address 127.0.0.1:33356)
0929 14:42:35.451126 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.451138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.451207 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.451649 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.451651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.451667 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.451740 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.451746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.453053 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.453056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.454222 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.454231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455183 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.455285 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455329 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.455445 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.455562 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"mutex_wait_us":28,"thread_start_us":77,"threads_started":1}
I20260929 14:42:35.455734 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.450812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.98.2:34695)
0929 14:42:35.451171 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.451180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.451200 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:42:35.451340 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.451346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.451529 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.451679 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.451825 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.452955 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.454342 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455048 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455166 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455330 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.455491 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.455588 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":97,"threads_started":1}
I20260929 14:42:35.455912 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.450375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.0.1:46006)
0929 14:42:35.451315 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.451327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.451427 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.451856 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.451858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.451882 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.451959 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.451965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.452887 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.452891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.454543 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.454549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455555 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.455642 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455661 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.455723 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.455784 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":201,"thread_start_us":90,"threads_started":1}
I20260929 14:42:35.456226 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.450535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.0.98.3:33987)
0929 14:42:35.450929 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.450940 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.451427 (+ 487us) server_negotiation.cc:407] Connection header received
0929 14:42:35.451526 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.451531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.451709 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.451855 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.452040 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.452754 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.454677 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455253 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455347 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455500 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.455993 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.456096 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
I20260929 14:42:35.456233 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.449795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.0.98.1:32931)
0929 14:42:35.450090 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.450098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.450472 (+ 374us) server_negotiation.cc:407] Connection header received
0929 14:42:35.450761 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.450768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.451009 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.451134 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.451603 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.452808 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.454672 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455410 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455506 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455648 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.456008 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.456109 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260929 14:42:35.457096 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.449669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32931 (local address 127.0.0.1:43926)
0929 14:42:35.450332 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.450381 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.450580 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.451332 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.451337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.451359 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.451480 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.451486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.452946 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.452949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.454537 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.454548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.455710 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.455713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.455799 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.455891 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.455961 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.456030 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":179,"thread_start_us":96,"threads_started":1}
I20260929 14:42:35.459208 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 04cc392d0fde44a980fb9a623051a93c (DEFAULT_TABLE table=test-table [id=27800d94ce4d4f628e119b593f0e4509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.459213 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 04cc392d0fde44a980fb9a623051a93c (DEFAULT_TABLE table=test-table [id=27800d94ce4d4f628e119b593f0e4509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.459220 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 04cc392d0fde44a980fb9a623051a93c (DEFAULT_TABLE table=test-table [id=27800d94ce4d4f628e119b593f0e4509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.460106 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cc392d0fde44a980fb9a623051a93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.460194 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cc392d0fde44a980fb9a623051a93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.460341 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04cc392d0fde44a980fb9a623051a93c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.466661 8431 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Bootstrap starting.
I20260929 14:42:35.468259 8433 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Bootstrap starting.
I20260929 14:42:35.468415 8432 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Bootstrap starting.
I20260929 14:42:35.468989 8431 tablet_bootstrap.cc:654] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.469691 8431 log.cc:824] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:35.470331 8432 tablet_bootstrap.cc:654] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.470331 8433 tablet_bootstrap.cc:654] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.471135 8432 log.cc:824] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:35.471136 8433 log.cc:824] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:35.471632 8431 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: No bootstrap required, opened a new log
I20260929 14:42:35.471825 8431 ts_tablet_manager.cc:1397] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 14:42:35.473136 8433 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: No bootstrap required, opened a new log
I20260929 14:42:35.473336 8433 ts_tablet_manager.cc:1397] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:42:35.473392 8432 tablet_bootstrap.cc:492] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: No bootstrap required, opened a new log
I20260929 14:42:35.473548 8432 ts_tablet_manager.cc:1397] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:42:35.475155 8431 raft_consensus.cc:348] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.475378 8431 raft_consensus.cc:374] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.475436 8431 raft_consensus.cc:729] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.475880 8431 consensus_queue.cc:260] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.476511 8123 heartbeater.cc:503] Master 127.0.98.62:41621 was elected leader, sending a full tablet report...
I20260929 14:42:35.476787 8431 ts_tablet_manager.cc:1428] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260929 14:42:35.477208 8433 raft_consensus.cc:348] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.477409 8433 raft_consensus.cc:374] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.477450 8433 raft_consensus.cc:729] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed413db696e4ee69138448595f5ac64, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.477583 8432 raft_consensus.cc:348] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.477795 8432 raft_consensus.cc:374] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.477869 8432 raft_consensus.cc:729] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.477855 8433 consensus_queue.cc:260] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.478433 8432 consensus_queue.cc:260] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.479005 8403 heartbeater.cc:503] Master 127.0.98.62:41621 was elected leader, sending a full tablet report...
I20260929 14:42:35.479375 8432 ts_tablet_manager.cc:1428] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260929 14:42:35.479704 8263 heartbeater.cc:503] Master 127.0.98.62:41621 was elected leader, sending a full tablet report...
I20260929 14:42:35.479789 8433 ts_tablet_manager.cc:1428] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:42:35.483079 8438 raft_consensus.cc:482] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:35.483225 8438 raft_consensus.cc:504] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.484225 8438 leader_election.cc:288] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695)
I20260929 14:42:35.488781 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.484399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32931 (local address 127.0.98.3:47907)
0929 14:42:35.484496 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.484511 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.484586 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.485044 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.485046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.485060 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.485157 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.485163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.486181 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.486184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.487519 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.487530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.488355 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.488359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.488466 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.488487 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.488578 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.488652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":2}
I20260929 14:42:35.488821 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.484641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47907 (local address 127.0.98.1:32931)
0929 14:42:35.484749 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.484754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.484767 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:35.484811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.484814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.484855 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.484946 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.485246 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.486032 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.487654 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.488145 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.488237 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.488299 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.488622 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.488703 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:42:35.489097 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.484805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58797 (local address 127.0.98.2:34695)
0929 14:42:35.484965 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.484969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.484980 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:35.485017 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.485019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.485053 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.485124 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.485363 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.486079 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.487939 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.488499 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.488597 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.488649 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.488835 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.488937 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:42:35.489130 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.484452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34695 (local address 127.0.98.3:58797)
0929 14:42:35.484747 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.484759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.484829 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.485149 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.485151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.485164 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.485265 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.485271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.486176 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.486180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.487810 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.487819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.488612 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.488615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.488715 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.488739 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.488812 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.488906 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":19,"thread_start_us":118,"threads_started":1}
I20260929 14:42:35.489457 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cc392d0fde44a980fb9a623051a93c" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604" is_pre_election: true
I20260929 14:42:35.489773 8077 raft_consensus.cc:2454] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 0.
I20260929 14:42:35.490247 8289 leader_election.cc:302] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, eed413db696e4ee69138448595f5ac64; no voters:
I20260929 14:42:35.490340 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cc392d0fde44a980fb9a623051a93c" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" is_pre_election: true
I20260929 14:42:35.490568 8438 raft_consensus.cc:2781] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:35.490661 8438 raft_consensus.cc:482] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:35.490640 8217 raft_consensus.cc:2454] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 0.
I20260929 14:42:35.490746 8438 raft_consensus.cc:3037] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.491735 8438 raft_consensus.cc:504] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.492067 8438 leader_election.cc:288] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 election: Requested vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695)
I20260929 14:42:35.492435 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cc392d0fde44a980fb9a623051a93c" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
I20260929 14:42:35.492458 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04cc392d0fde44a980fb9a623051a93c" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604"
I20260929 14:42:35.492566 8217 raft_consensus.cc:3037] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.492576 8077 raft_consensus.cc:3037] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.493856 8077 raft_consensus.cc:2454] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 1.
I20260929 14:42:35.493856 8217 raft_consensus.cc:2454] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 1.
I20260929 14:42:35.494166 8289 leader_election.cc:302] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, eed413db696e4ee69138448595f5ac64; no voters:
I20260929 14:42:35.494325 8438 raft_consensus.cc:2781] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:35.494565 8438 raft_consensus.cc:686] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 1 LEADER]: Becoming Leader. State: Replica: eed413db696e4ee69138448595f5ac64, State: Running, Role: LEADER
I20260929 14:42:35.494865 8438 consensus_queue.cc:237] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.498283 7919 catalog_manager.cc:5495] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed413db696e4ee69138448595f5ac64 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: HEALTHY } } }
I20260929 14:42:35.519259 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.515776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.98.3:33987)
0929 14:42:35.515889 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.515892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.515906 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:35.516004 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.516006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.516060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.516149 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:35.516400 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.517023 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.517843 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.518125 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.518206 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.518460 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.518883 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.519044 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.519096 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.519150 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:42:35.525636 8217 raft_consensus.cc:1261] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Refusing update from remote peer eed413db696e4ee69138448595f5ac64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.526288 8077 raft_consensus.cc:1261] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Refusing update from remote peer eed413db696e4ee69138448595f5ac64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.526439 8438 consensus_queue.cc:1035] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.526912 8441 consensus_queue.cc:1035] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.535511 8446 mvcc.cc:205] Tried to move back new op lower bound from 7334678345824632832 to 7334678345709543424. Current Snapshot: MvccSnapshot[applied={T|T < 7334678345824632832}]
I20260929 14:42:35.543401 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:35.543570 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:35.546212 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Sending DeleteTablet for 3 replicas of tablet 04cc392d0fde44a980fb9a623051a93c
I20260929 14:42:35.547166 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cc392d0fde44a980fb9a623051a93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260929 14:42:35.547441 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cc392d0fde44a980fb9a623051a93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260929 14:42:35.547762 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 04cc392d0fde44a980fb9a623051a93c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46006
I20260929 14:42:35.547801 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:42:35.547992 8454 tablet_replica.cc:330] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: stopping tablet replica
I20260929 14:42:35.548329 8454 raft_consensus.cc:2229] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 14:42:35.548353 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:35.548421 8453 tablet_replica.cc:330] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: stopping tablet replica
I20260929 14:42:35.548704 8454 raft_consensus.cc:2258] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.548776 8453 raft_consensus.cc:2229] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:35.549106 8453 raft_consensus.cc:2258] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.549471 8455 tablet_replica.cc:330] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: stopping tablet replica
I20260929 14:42:35.549808 8455 raft_consensus.cc:2229] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:35.550288 8455 raft_consensus.cc:2258] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.551608 8455 ts_tablet_manager.cc:1905] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.551652 8453 ts_tablet_manager.cc:1905] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.554525 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 (DEFAULT_TABLE table=test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.554800 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd8baa2f2d74d56b52deca2df8c3308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.554798 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 (DEFAULT_TABLE table=test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.554966 8453 ts_tablet_manager.cc:1918] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:35.555076 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd8baa2f2d74d56b52deca2df8c3308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.555107 8453 log.cc:1196] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/04cc392d0fde44a980fb9a623051a93c
I20260929 14:42:35.555526 8453 ts_tablet_manager.cc:1939] T 04cc392d0fde44a980fb9a623051a93c P e7c3383dc73d4c35b786f53b97435604: Deleting consensus metadata
I20260929 14:42:35.557005 7905 catalog_manager.cc:4841] TS e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931): tablet 04cc392d0fde44a980fb9a623051a93c (table test-table [id=27800d94ce4d4f628e119b593f0e4509]) successfully deleted
I20260929 14:42:35.557250 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 (DEFAULT_TABLE table=test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.557514 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd8baa2f2d74d56b52deca2df8c3308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.557543 8432 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Bootstrap starting.
I20260929 14:42:35.558346 8432 tablet_bootstrap.cc:654] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.559612 8432 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: No bootstrap required, opened a new log
I20260929 14:42:35.559697 8432 ts_tablet_manager.cc:1397] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:42:35.559710 8433 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Bootstrap starting.
I20260929 14:42:35.559825 8431 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Bootstrap starting.
I20260929 14:42:35.560170 8432 raft_consensus.cc:348] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.560280 8432 raft_consensus.cc:374] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.560312 8432 raft_consensus.cc:729] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.560437 8432 consensus_queue.cc:260] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.560628 8433 tablet_bootstrap.cc:654] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.560655 8431 tablet_bootstrap.cc:654] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.560709 8432 ts_tablet_manager.cc:1428] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:35.560794 8454 ts_tablet_manager.cc:1905] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.561980 8455 ts_tablet_manager.cc:1918] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:35.562083 8455 log.cc:1196] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/04cc392d0fde44a980fb9a623051a93c
I20260929 14:42:35.562371 8455 ts_tablet_manager.cc:1939] T 04cc392d0fde44a980fb9a623051a93c P eed413db696e4ee69138448595f5ac64: Deleting consensus metadata
I20260929 14:42:35.563623 7906 catalog_manager.cc:4841] TS eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987): tablet 04cc392d0fde44a980fb9a623051a93c (table test-table [id=27800d94ce4d4f628e119b593f0e4509]) successfully deleted
I20260929 14:42:35.564065 8454 ts_tablet_manager.cc:1918] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:35.564159 8454 log.cc:1196] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/04cc392d0fde44a980fb9a623051a93c
I20260929 14:42:35.564438 8454 ts_tablet_manager.cc:1939] T 04cc392d0fde44a980fb9a623051a93c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting consensus metadata
I20260929 14:42:35.564544 8431 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: No bootstrap required, opened a new log
I20260929 14:42:35.564658 8431 ts_tablet_manager.cc:1397] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 14:42:35.565326 8431 raft_consensus.cc:348] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.565451 8431 raft_consensus.cc:374] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.565481 8431 raft_consensus.cc:729] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.565603 8431 consensus_queue.cc:260] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.565850 8431 ts_tablet_manager.cc:1428] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:35.566061 7906 catalog_manager.cc:4841] TS fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695): tablet 04cc392d0fde44a980fb9a623051a93c (table test-table [id=27800d94ce4d4f628e119b593f0e4509]) successfully deleted
I20260929 14:42:35.567220 8433 tablet_bootstrap.cc:492] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: No bootstrap required, opened a new log
I20260929 14:42:35.567339 8433 ts_tablet_manager.cc:1397] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260929 14:42:35.567806 8433 raft_consensus.cc:348] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.567941 8433 raft_consensus.cc:374] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.567976 8433 raft_consensus.cc:729] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed413db696e4ee69138448595f5ac64, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.568104 8433 consensus_queue.cc:260] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.568362 8433 ts_tablet_manager.cc:1428] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:42:35.605849 8437 raft_consensus.cc:482] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:35.606022 8437 raft_consensus.cc:504] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.607329 8437 leader_election.cc:288] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:35.612032 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.607666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35737 (local address 127.0.98.2:34695)
0929 14:42:35.607806 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.607811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.607973 (+ 162us) server_negotiation.cc:407] Connection header received
0929 14:42:35.608174 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.608177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.608232 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.608283 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.608745 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.609398 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.610799 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.611420 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.611530 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.611660 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.611814 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.611899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:42:35.612062 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.607608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34695 (local address 127.0.98.1:35737)
0929 14:42:35.607885 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.607900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.608023 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.608466 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.608469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.608487 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.608604 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.608610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.609500 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.609503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.610649 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.610660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.611584 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.611588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.611693 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.611718 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.611793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.611873 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":22,"thread_start_us":72,"threads_started":1}
I20260929 14:42:35.612286 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.608188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57105 (local address 127.0.98.3:33987)
0929 14:42:35.608311 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.608315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.608570 (+ 255us) server_negotiation.cc:407] Connection header received
0929 14:42:35.608661 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.608664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.608712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.608800 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:35.609097 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.609845 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.611123 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.611554 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.611678 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.611766 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.612093 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.612176 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:42:35.612457 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd8baa2f2d74d56b52deca2df8c3308" candidate_uuid: "e7c3383dc73d4c35b786f53b97435604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" is_pre_election: true
I20260929 14:42:35.612605 8217 raft_consensus.cc:2454] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c3383dc73d4c35b786f53b97435604 in term 0.
I20260929 14:42:35.612651 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.608097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.98.1:57105)
0929 14:42:35.608484 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.608497 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.608580 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.608817 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.608820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.608839 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:35.608985 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.608991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.609981 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.609984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.611009 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.611018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.611846 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.611850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.611938 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.611961 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.612422 (+ 461us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.612505 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260929 14:42:35.613164 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd8baa2f2d74d56b52deca2df8c3308" candidate_uuid: "e7c3383dc73d4c35b786f53b97435604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64" is_pre_election: true
I20260929 14:42:35.613214 8010 leader_election.cc:302] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:35.613401 8356 raft_consensus.cc:2454] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c3383dc73d4c35b786f53b97435604 in term 0.
I20260929 14:42:35.613512 8437 raft_consensus.cc:2781] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:35.613564 8437 raft_consensus.cc:482] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:35.613606 8437 raft_consensus.cc:3037] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.614703 8437 raft_consensus.cc:504] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.615028 8437 leader_election.cc:288] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [CANDIDATE]: Term 1 election: Requested vote from peers fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:35.615430 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd8baa2f2d74d56b52deca2df8c3308" candidate_uuid: "e7c3383dc73d4c35b786f53b97435604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64"
I20260929 14:42:35.615384 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd8baa2f2d74d56b52deca2df8c3308" candidate_uuid: "e7c3383dc73d4c35b786f53b97435604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
I20260929 14:42:35.615542 8356 raft_consensus.cc:3037] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.615542 8217 raft_consensus.cc:3037] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.616690 8217 raft_consensus.cc:2454] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c3383dc73d4c35b786f53b97435604 in term 1.
I20260929 14:42:35.616683 8356 raft_consensus.cc:2454] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c3383dc73d4c35b786f53b97435604 in term 1.
I20260929 14:42:35.617012 8010 leader_election.cc:302] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:35.617295 8437 raft_consensus.cc:2781] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:35.617515 8437 raft_consensus.cc:686] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 1 LEADER]: Becoming Leader. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Running, Role: LEADER
I20260929 14:42:35.617774 8437 consensus_queue.cc:237] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.620398 7919 catalog_manager.cc:5495] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7c3383dc73d4c35b786f53b97435604 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e7c3383dc73d4c35b786f53b97435604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: UNKNOWN } } }
W20260929 14:42:35.637030 8404 tablet.cc:2369] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692954, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:35.669505 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.658853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41621 (local address 127.0.0.1:43456)
0929 14:42:35.659273 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.659288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.659426 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.659705 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.659708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.660225 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:35.660347 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.660355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.661437 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.661440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.662305 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.662317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.662923 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.662927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.663016 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.663679 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:35.663693 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:35.665675 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:35.667617 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.667624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.667631 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.667962 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.668265 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:35.668268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:35.668270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:35.668354 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:35.668808 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:35.668815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:35.668982 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.669103 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.669274 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260929 14:42:35.669534 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.658950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.98.62:41621)
0929 14:42:35.659103 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.659107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.659382 (+ 275us) server_negotiation.cc:407] Connection header received
0929 14:42:35.659527 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.659530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.659577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.659659 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.660470 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.661310 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.662435 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.662776 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.662875 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.665842 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.665847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.665850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.665886 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.667496 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.668075 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.668079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.668080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.668173 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.668449 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.668453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.668454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.668700 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.668808 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.669180 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.669355 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":75}
I20260929 14:42:35.674595 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 14:42:35.675390 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260929 14:42:35.678615 8264 tablet.cc:2369] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:35.680429 8057 tablet_service.cc:1467] Processing CreateTablet for tablet efab56b0ce474e22b0fe841e2db60827 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5dcfb46e842739097b623dbed014c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:35.680493 8337 tablet_service.cc:1467] Processing CreateTablet for tablet efab56b0ce474e22b0fe841e2db60827 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5dcfb46e842739097b623dbed014c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:35.680831 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efab56b0ce474e22b0fe841e2db60827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.681037 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efab56b0ce474e22b0fe841e2db60827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.683838 8197 tablet_service.cc:1467] Processing CreateTablet for tablet efab56b0ce474e22b0fe841e2db60827 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91e5dcfb46e842739097b623dbed014c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 14:42:35.684130 8433 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: Bootstrap starting.
I20260929 14:42:35.684181 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efab56b0ce474e22b0fe841e2db60827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.685084 8433 tablet_bootstrap.cc:654] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.686417 8433 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: No bootstrap required, opened a new log
I20260929 14:42:35.686499 8433 ts_tablet_manager.cc:1397] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 14:42:35.686750 8432 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: Bootstrap starting.
I20260929 14:42:35.686918 8433 raft_consensus.cc:348] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.687021 8433 raft_consensus.cc:374] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.687047 8433 raft_consensus.cc:729] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed413db696e4ee69138448595f5ac64, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.687142 8431 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: Bootstrap starting.
I20260929 14:42:35.687157 8433 consensus_queue.cc:260] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.687391 8433 ts_tablet_manager.cc:1428] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:35.687552 8438 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.687655 8438 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.687690 8432 tablet_bootstrap.cc:654] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.688110 8431 tablet_bootstrap.cc:654] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.689792 8432 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: No bootstrap required, opened a new log
I20260929 14:42:35.689893 8432 ts_tablet_manager.cc:1397] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 14:42:35.690049 8431 tablet_bootstrap.cc:492] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: No bootstrap required, opened a new log
I20260929 14:42:35.690153 8431 ts_tablet_manager.cc:1397] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:35.690383 8432 raft_consensus.cc:348] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.690514 8432 raft_consensus.cc:374] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.690557 8432 raft_consensus.cc:729] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.690558 8431 raft_consensus.cc:348] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.690665 8431 raft_consensus.cc:374] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.690701 8431 raft_consensus.cc:729] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.690729 8432 consensus_queue.cc:260] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.690814 8431 consensus_queue.cc:260] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.691076 8432 ts_tablet_manager.cc:1428] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260929 14:42:35.691075 8431 ts_tablet_manager.cc:1428] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:42:35.691015 8439 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.691186 8439 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.691380 8437 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.691493 8437 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 14:42:35.696702 8124 tablet.cc:2369] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:35.752465 8438 raft_consensus.cc:482] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:35.752599 8438 raft_consensus.cc:504] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.753126 8438 leader_election.cc:288] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695)
I20260929 14:42:35.753516 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efab56b0ce474e22b0fe841e2db60827" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" is_pre_election: true
I20260929 14:42:35.753543 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efab56b0ce474e22b0fe841e2db60827" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604" is_pre_election: true
I20260929 14:42:35.753647 8217 raft_consensus.cc:2454] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 0.
I20260929 14:42:35.753695 8077 raft_consensus.cc:2454] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 0.
I20260929 14:42:35.753942 8290 leader_election.cc:302] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eed413db696e4ee69138448595f5ac64, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:35.754108 8438 raft_consensus.cc:2781] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:35.754168 8438 raft_consensus.cc:482] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:35.754194 8438 raft_consensus.cc:3037] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.755195 8438 raft_consensus.cc:504] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.755527 8438 leader_election.cc:288] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 election: Requested vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695)
I20260929 14:42:35.755928 8217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efab56b0ce474e22b0fe841e2db60827" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988"
I20260929 14:42:35.756081 8217 raft_consensus.cc:3037] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.755928 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efab56b0ce474e22b0fe841e2db60827" candidate_uuid: "eed413db696e4ee69138448595f5ac64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604"
I20260929 14:42:35.756165 8077 raft_consensus.cc:3037] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:35.757261 8077 raft_consensus.cc:2454] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 1.
I20260929 14:42:35.757261 8217 raft_consensus.cc:2454] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eed413db696e4ee69138448595f5ac64 in term 1.
I20260929 14:42:35.757563 8290 leader_election.cc:302] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eed413db696e4ee69138448595f5ac64, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:35.757720 8438 raft_consensus.cc:2781] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:35.757807 8438 raft_consensus.cc:686] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [term 1 LEADER]: Becoming Leader. State: Replica: eed413db696e4ee69138448595f5ac64, State: Running, Role: LEADER
I20260929 14:42:35.757941 8438 consensus_queue.cc:237] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.758627 8441 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: TxnStatusTablet state changed. Reason: New leader eed413db696e4ee69138448595f5ac64. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.758726 8441 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:35.758981 8473 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 14:42:35.759146 8473 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 14:42:35.759670 7919 catalog_manager.cc:5495] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 reported cstate change: term changed from 0 to 1, leader changed from <none> to eed413db696e4ee69138448595f5ac64 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: HEALTHY } } }
I20260929 14:42:35.774403 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.770939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.0.98.3:33987)
0929 14:42:35.771042 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.771045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.771057 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:35.771213 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.771215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.771256 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.771316 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:35.771567 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.772181 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.773381 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.773619 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.773699 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.773984 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.774059 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.774115 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.774237 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.774283 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 14:42:35.774430 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.770823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.0.1:46038)
0929 14:42:35.770989 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.771005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.771106 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.771343 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.771346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.771354 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:35.771459 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.771465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.772278 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.772281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.773270 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.773281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.773714 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.773717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.773806 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.773859 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.774133 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.774144 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.774207 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.774289 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 14:42:35.777068 8217 raft_consensus.cc:1261] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Refusing update from remote peer eed413db696e4ee69138448595f5ac64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.777074 8077 raft_consensus.cc:1261] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Refusing update from remote peer eed413db696e4ee69138448595f5ac64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.777453 8441 consensus_queue.cc:1035] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.777626 8438 consensus_queue.cc:1035] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.779073 8439 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: TxnStatusTablet state changed. Reason: New leader eed413db696e4ee69138448595f5ac64. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.779193 8439 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.779287 8437 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: TxnStatusTablet state changed. Reason: New leader eed413db696e4ee69138448595f5ac64. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.779388 8437 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.780297 8444 mvcc.cc:205] Tried to move back new op lower bound from 7334678346858549248 to 7334678346785595392. Current Snapshot: MvccSnapshot[applied={T|T < 7334678346858549248}]
I20260929 14:42:35.780411 8441 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.780516 8441 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:35.781876 8445 mvcc.cc:205] Tried to move back new op lower bound from 7334678346858549248 to 7334678346785595392. Current Snapshot: MvccSnapshot[applied={T|T < 7334678346858549248 or (T in {7334678346858549248})}]
I20260929 14:42:35.782112 8437 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.782199 8437 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P e7c3383dc73d4c35b786f53b97435604: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.786154 8441 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.786270 8441 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P eed413db696e4ee69138448595f5ac64: This TxnStatusTablet replica's current role is: LEADER
I20260929 14:42:35.787091 8439 tablet_replica.cc:439] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eed413db696e4ee69138448595f5ac64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } } }
I20260929 14:42:35.787204 8439 tablet_replica.cc:442] T efab56b0ce474e22b0fe841e2db60827 P fb4d8c90759a4c8eac8d4fbb7834d988: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 14:42:35.789861 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.785078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.98.1:32931)
0929 14:42:35.785178 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.785182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.785238 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:42:35.785390 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.785392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.785439 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.785534 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:35.785773 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.786455 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.787898 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.788205 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.788298 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.788990 (+ 692us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.789481 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.789587 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.789654 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.789715 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:42:35.800266 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.797040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45351 (local address 127.0.98.3:33987)
0929 14:42:35.797167 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.797171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.797299 (+ 128us) server_negotiation.cc:407] Connection header received
0929 14:42:35.797465 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.797468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.797509 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.797572 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:35.797819 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.798314 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.799265 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.799481 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.799576 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.799847 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.799928 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.799987 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.800081 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.800127 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:42:35.800519 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.796926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.98.1:45351)
0929 14:42:35.797219 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.797232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.797333 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.797590 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.797593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.797600 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:35.797713 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.797718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.798412 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.798416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.799143 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.799154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.799589 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.799592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.799684 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.799724 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.800004 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.800011 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.800070 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.800146 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
I20260929 14:42:35.807351 8217 raft_consensus.cc:1261] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Refusing update from remote peer e7c3383dc73d4c35b786f53b97435604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.807473 8356 raft_consensus.cc:1261] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Refusing update from remote peer e7c3383dc73d4c35b786f53b97435604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:35.807878 8437 consensus_queue.cc:1035] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.808075 8461 consensus_queue.cc:1035] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:35.832578 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.828628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58733 (local address 127.0.98.1:32931)
0929 14:42:35.829087 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.829091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.829104 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:35.829143 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.829145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.829184 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.829247 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:35.829579 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.830251 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.831450 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.831789 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.831897 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.832186 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.832253 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.832302 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.832391 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.832470 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260929 14:42:35.832616 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.828546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32931 (local address 127.0.98.3:58733)
0929 14:42:35.828805 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:35.828821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:35.828932 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:35.829357 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:35.829360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:35.829368 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:35.829464 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.829469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.830348 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.830351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.831308 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:35.831317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.831913 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.831917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:35.832009 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.832068 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:35.832319 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:35.832325 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:35.832382 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:42:35.832466 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260929 14:42:35.857508 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:35.857671 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:35.859720 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Sending DeleteTablet for 3 replicas of tablet 6cd8baa2f2d74d56b52deca2df8c3308
I20260929 14:42:35.860291 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260929 14:42:35.860426 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46006
I20260929 14:42:35.860493 8453 tablet_replica.cc:330] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: stopping tablet replica
I20260929 14:42:35.860581 8455 tablet_replica.cc:330] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: stopping tablet replica
I20260929 14:42:35.860596 8453 raft_consensus.cc:2229] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:35.860672 8455 raft_consensus.cc:2229] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:35.860791 8455 raft_consensus.cc:2258] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.861167 8453 raft_consensus.cc:2258] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.861734 8455 ts_tablet_manager.cc:1905] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.861927 8453 ts_tablet_manager.cc:1905] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.864632 8453 ts_tablet_manager.cc:1918] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:35.864704 8453 log.cc:1196] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/6cd8baa2f2d74d56b52deca2df8c3308
I20260929 14:42:35.864956 8453 ts_tablet_manager.cc:1939] T 6cd8baa2f2d74d56b52deca2df8c3308 P e7c3383dc73d4c35b786f53b97435604: Deleting consensus metadata
I20260929 14:42:35.865460 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd8baa2f2d74d56b52deca2df8c3308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:35 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260929 14:42:35.865653 8454 tablet_replica.cc:330] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: stopping tablet replica
I20260929 14:42:35.865763 8454 raft_consensus.cc:2229] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:35.865907 8454 raft_consensus.cc:2258] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:35.866179 8455 ts_tablet_manager.cc:1918] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:35.866248 8455 log.cc:1196] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/6cd8baa2f2d74d56b52deca2df8c3308
I20260929 14:42:35.866472 8455 ts_tablet_manager.cc:1939] T 6cd8baa2f2d74d56b52deca2df8c3308 P eed413db696e4ee69138448595f5ac64: Deleting consensus metadata
I20260929 14:42:35.867012 8454 ts_tablet_manager.cc:1905] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:35.867202 7906 catalog_manager.cc:4841] TS eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987): tablet 6cd8baa2f2d74d56b52deca2df8c3308 (table test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]) successfully deleted
I20260929 14:42:35.869341 8454 ts_tablet_manager.cc:1918] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:35.869405 8454 log.cc:1196] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/6cd8baa2f2d74d56b52deca2df8c3308
I20260929 14:42:35.869555 8454 ts_tablet_manager.cc:1939] T 6cd8baa2f2d74d56b52deca2df8c3308 P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting consensus metadata
I20260929 14:42:35.870415 7906 catalog_manager.cc:4841] TS fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695): tablet 6cd8baa2f2d74d56b52deca2df8c3308 (table test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]) successfully deleted
I20260929 14:42:35.874169 7905 catalog_manager.cc:4841] TS e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931): tablet 6cd8baa2f2d74d56b52deca2df8c3308 (table test-table [id=c4f6f59aab84428c98bb1c0eac2d77c9]) successfully deleted
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:42:35.905241 7969 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:35.897366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.0.98.62:41621)
0929 14:42:35.897494 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.897498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.897786 (+ 288us) server_negotiation.cc:407] Connection header received
0929 14:42:35.897918 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.897922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.897981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.898073 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.898893 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.899734 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.901023 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.901378 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.901486 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.905104 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43472: BlockingRecv error: recv got EOF from 127.0.0.1:43472 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 14:42:35 dist-test-slave-g8hp krb5kdc[7875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 14:42:35.917519 7969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:35.907243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.98.62:41621)
0929 14:42:35.907356 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:35.907360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:35.907764 (+ 404us) server_negotiation.cc:407] Connection header received
0929 14:42:35.907893 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:35.907897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:35.907948 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:35.908028 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:35.908737 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.909617 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.910770 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:35.911112 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:35.911212 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:35.914145 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:35.914149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:35.914152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:35.914184 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:35.915778 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.916281 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.916284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.916286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.916373 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:35.916619 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:35.916622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:35.916623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:35.916830 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:35.916998 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:35.917219 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 14:42:35.917340 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:42:35.920517 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:35.921010 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:35.925668 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 5f667d7690f94f49b98628026d415d8c (DEFAULT_TABLE table=test-table [id=cce657e78b964ef08a3cae496b24c880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.925762 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 5f667d7690f94f49b98628026d415d8c (DEFAULT_TABLE table=test-table [id=cce657e78b964ef08a3cae496b24c880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.925930 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f667d7690f94f49b98628026d415d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.926040 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f667d7690f94f49b98628026d415d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.927930 8431 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Bootstrap starting.
I20260929 14:42:35.928360 8432 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Bootstrap starting.
I20260929 14:42:35.928910 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 5f667d7690f94f49b98628026d415d8c (DEFAULT_TABLE table=test-table [id=cce657e78b964ef08a3cae496b24c880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:35.929181 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f667d7690f94f49b98628026d415d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:35.929242 8431 tablet_bootstrap.cc:654] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.929268 8432 tablet_bootstrap.cc:654] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.930876 8431 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: No bootstrap required, opened a new log
I20260929 14:42:35.930955 8431 ts_tablet_manager.cc:1397] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 14:42:35.931056 8432 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: No bootstrap required, opened a new log
I20260929 14:42:35.931123 8432 ts_tablet_manager.cc:1397] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:35.931406 8431 raft_consensus.cc:348] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.931514 8431 raft_consensus.cc:374] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.931547 8431 raft_consensus.cc:729] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.931512 8432 raft_consensus.cc:348] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.931603 8432 raft_consensus.cc:374] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.931632 8432 raft_consensus.cc:729] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.931668 8431 consensus_queue.cc:260] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.931731 8432 consensus_queue.cc:260] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.931927 8431 ts_tablet_manager.cc:1428] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:35.931958 8432 ts_tablet_manager.cc:1428] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:35.932148 8433 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Bootstrap starting.
I20260929 14:42:35.932989 8433 tablet_bootstrap.cc:654] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:35.934171 8433 tablet_bootstrap.cc:492] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: No bootstrap required, opened a new log
I20260929 14:42:35.934250 8433 ts_tablet_manager.cc:1397] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260929 14:42:35.934644 8433 raft_consensus.cc:348] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.934765 8433 raft_consensus.cc:374] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:35.934806 8433 raft_consensus.cc:729] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed413db696e4ee69138448595f5ac64, State: Initialized, Role: FOLLOWER
I20260929 14:42:35.934929 8433 consensus_queue.cc:260] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:35.935182 8433 ts_tablet_manager.cc:1428] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:42:36.201346 8439 raft_consensus.cc:482] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:36.201489 8439 raft_consensus.cc:504] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.202598 8439 leader_election.cc:288] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:36.206390 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.202962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60937 (local address 127.0.98.1:32931)
0929 14:42:36.203095 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:36.203099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:36.203302 (+ 203us) server_negotiation.cc:407] Connection header received
0929 14:42:36.203451 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:36.203454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:36.203494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:36.203556 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:36.203793 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.204338 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.205439 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.205883 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.205974 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.206042 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:36.206202 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 14:42:36.206267 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:42:36.206446 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.202837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32931 (local address 127.0.98.2:60937)
0929 14:42:36.203197 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:36.203211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:36.203336 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:36.203575 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:36.203578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:36.203597 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:36.203696 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.203702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.204445 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.204448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.205325 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.205334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.206009 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.206013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.206094 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.206115 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:42:36.206178 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:42:36.206248 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260929 14:42:36.206457 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.203016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56519 (local address 127.0.98.3:33987)
0929 14:42:36.203137 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:36.203141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:36.203389 (+ 248us) server_negotiation.cc:407] Connection header received
0929 14:42:36.203568 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:36.203570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:36.203621 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:36.203683 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:36.203903 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.204382 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.205448 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.205883 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.205974 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.206040 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:36.206274 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 14:42:36.206342 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 14:42:36.206471 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.202900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.98.2:56519)
0929 14:42:36.203306 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:36.203319 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:36.203418 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:36.203709 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:36.203711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:36.203727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:36.203811 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.203817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.204476 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.204479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.205333 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.205342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.206017 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.206021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.206160 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.206178 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:42:36.206250 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 14:42:36.206314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":66,"threads_started":1}
I20260929 14:42:36.206933 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f667d7690f94f49b98628026d415d8c" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64" is_pre_election: true
I20260929 14:42:36.206995 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f667d7690f94f49b98628026d415d8c" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604" is_pre_election: true
I20260929 14:42:36.207079 8356 raft_consensus.cc:2454] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 0.
I20260929 14:42:36.207139 8077 raft_consensus.cc:2454] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 0.
I20260929 14:42:36.207501 8149 leader_election.cc:302] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:36.207772 8439 raft_consensus.cc:2781] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:36.207825 8439 raft_consensus.cc:482] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:36.207851 8439 raft_consensus.cc:3037] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.208835 8439 raft_consensus.cc:504] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.209223 8439 leader_election.cc:288] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 election: Requested vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:36.209619 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f667d7690f94f49b98628026d415d8c" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64"
I20260929 14:42:36.209647 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f667d7690f94f49b98628026d415d8c" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604"
I20260929 14:42:36.209740 8356 raft_consensus.cc:3037] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.209753 8077 raft_consensus.cc:3037] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.210623 8077 raft_consensus.cc:2454] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 1.
I20260929 14:42:36.210659 8356 raft_consensus.cc:2454] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 1.
I20260929 14:42:36.210894 8149 leader_election.cc:302] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:36.211062 8439 raft_consensus.cc:2781] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:36.211228 8439 raft_consensus.cc:686] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 LEADER]: Becoming Leader. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Running, Role: LEADER
I20260929 14:42:36.211457 8439 consensus_queue.cc:237] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.213740 7918 catalog_manager.cc:5495] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:36.233253 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.229504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.0.98.2:34695)
0929 14:42:36.229791 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:36.229797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:36.229810 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:36.229919 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:36.229922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:36.229977 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:36.230081 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:36.230385 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.230950 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.231835 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.232112 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.232194 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.232405 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:36.232920 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:36.233011 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:36.233066 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 14:42:36.233117 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260929 14:42:36.235985 8077 raft_consensus.cc:1261] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Refusing update from remote peer fb4d8c90759a4c8eac8d4fbb7834d988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:36.236064 8356 raft_consensus.cc:1261] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Refusing update from remote peer fb4d8c90759a4c8eac8d4fbb7834d988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:36.236450 8439 consensus_queue.cc:1035] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:36.236652 8504 consensus_queue.cc:1035] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:36.247098 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:36.247277 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:36.249372 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Sending DeleteTablet for 3 replicas of tablet 5f667d7690f94f49b98628026d415d8c
I20260929 14:42:36.249975 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f667d7690f94f49b98628026d415d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260929 14:42:36.250106 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f667d7690f94f49b98628026d415d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46006
I20260929 14:42:36.250324 8453 tablet_replica.cc:330] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: stopping tablet replica
I20260929 14:42:36.250466 8453 raft_consensus.cc:2229] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:36.250623 8453 raft_consensus.cc:2258] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:36.250818 8455 tablet_replica.cc:330] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: stopping tablet replica
I20260929 14:42:36.250828 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 14:42:36.250948 8455 raft_consensus.cc:2229] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:36.251066 8455 raft_consensus.cc:2258] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 14:42:36.251272 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:36.251441 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f667d7690f94f49b98628026d415d8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260929 14:42:36.251631 8454 tablet_replica.cc:330] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: stopping tablet replica
I20260929 14:42:36.251741 8454 raft_consensus.cc:2229] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:36.252049 8454 raft_consensus.cc:2258] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:36.252467 8455 ts_tablet_manager.cc:1905] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:36.252655 8454 ts_tablet_manager.cc:1905] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:36.252655 8453 ts_tablet_manager.cc:1905] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:36.255033 8453 ts_tablet_manager.cc:1918] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:36.255043 8454 ts_tablet_manager.cc:1918] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:36.255112 8454 log.cc:1196] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/5f667d7690f94f49b98628026d415d8c
I20260929 14:42:36.255112 8453 log.cc:1196] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/5f667d7690f94f49b98628026d415d8c
I20260929 14:42:36.255347 8454 ts_tablet_manager.cc:1939] T 5f667d7690f94f49b98628026d415d8c P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting consensus metadata
I20260929 14:42:36.255347 8453 ts_tablet_manager.cc:1939] T 5f667d7690f94f49b98628026d415d8c P e7c3383dc73d4c35b786f53b97435604: Deleting consensus metadata
I20260929 14:42:36.255600 8455 ts_tablet_manager.cc:1918] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:36.255693 8455 log.cc:1196] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/5f667d7690f94f49b98628026d415d8c
I20260929 14:42:36.255911 8455 ts_tablet_manager.cc:1939] T 5f667d7690f94f49b98628026d415d8c P eed413db696e4ee69138448595f5ac64: Deleting consensus metadata
I20260929 14:42:36.256285 7906 catalog_manager.cc:4841] TS fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695): tablet 5f667d7690f94f49b98628026d415d8c (table test-table [id=cce657e78b964ef08a3cae496b24c880]) successfully deleted
I20260929 14:42:36.256516 7906 catalog_manager.cc:4841] TS eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987): tablet 5f667d7690f94f49b98628026d415d8c (table test-table [id=cce657e78b964ef08a3cae496b24c880]) successfully deleted
I20260929 14:42:36.257817 8197 tablet_service.cc:1467] Processing CreateTablet for tablet 35939c614bd0455085a1965d2f01b58e (DEFAULT_TABLE table=test-table [id=94b1e36949ca43c588b11e0918333e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:36.257856 8057 tablet_service.cc:1467] Processing CreateTablet for tablet 35939c614bd0455085a1965d2f01b58e (DEFAULT_TABLE table=test-table [id=94b1e36949ca43c588b11e0918333e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:36.257856 8337 tablet_service.cc:1467] Processing CreateTablet for tablet 35939c614bd0455085a1965d2f01b58e (DEFAULT_TABLE table=test-table [id=94b1e36949ca43c588b11e0918333e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:36.258103 8337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35939c614bd0455085a1965d2f01b58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:36.258103 8057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35939c614bd0455085a1965d2f01b58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:36.258104 8197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35939c614bd0455085a1965d2f01b58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:36.260349 8433 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Bootstrap starting.
I20260929 14:42:36.260349 8431 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: Bootstrap starting.
I20260929 14:42:36.260358 8432 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Bootstrap starting.
I20260929 14:42:36.261209 8431 tablet_bootstrap.cc:654] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:36.261209 8433 tablet_bootstrap.cc:654] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:36.261296 8432 tablet_bootstrap.cc:654] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:36.261729 7905 catalog_manager.cc:4841] TS e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931): tablet 5f667d7690f94f49b98628026d415d8c (table test-table [id=cce657e78b964ef08a3cae496b24c880]) successfully deleted
I20260929 14:42:36.262502 8432 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: No bootstrap required, opened a new log
I20260929 14:42:36.262596 8432 ts_tablet_manager.cc:1397] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:36.263032 8431 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: No bootstrap required, opened a new log
I20260929 14:42:36.263036 8432 raft_consensus.cc:348] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.263113 8431 ts_tablet_manager.cc:1397] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:36.263163 8432 raft_consensus.cc:374] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:36.263200 8432 raft_consensus.cc:729] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Initialized, Role: FOLLOWER
I20260929 14:42:36.263325 8432 consensus_queue.cc:260] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.263500 8431 raft_consensus.cc:348] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.263589 8432 ts_tablet_manager.cc:1428] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:36.263605 8431 raft_consensus.cc:374] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:36.263643 8431 raft_consensus.cc:729] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c3383dc73d4c35b786f53b97435604, State: Initialized, Role: FOLLOWER
I20260929 14:42:36.263757 8431 consensus_queue.cc:260] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.264010 8431 ts_tablet_manager.cc:1428] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 14:42:36.265321 8433 tablet_bootstrap.cc:492] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: No bootstrap required, opened a new log
I20260929 14:42:36.265420 8433 ts_tablet_manager.cc:1397] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 14:42:36.265816 8433 raft_consensus.cc:348] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.265930 8433 raft_consensus.cc:374] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:36.265959 8433 raft_consensus.cc:729] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed413db696e4ee69138448595f5ac64, State: Initialized, Role: FOLLOWER
I20260929 14:42:36.266073 8433 consensus_queue.cc:260] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.266322 8433 ts_tablet_manager.cc:1428] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 14:42:36.389087 8511 raft_consensus.cc:482] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:36.389242 8511 raft_consensus.cc:504] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.389812 8511 leader_election.cc:288] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:36.390129 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35939c614bd0455085a1965d2f01b58e" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604" is_pre_election: true
I20260929 14:42:36.390198 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35939c614bd0455085a1965d2f01b58e" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64" is_pre_election: true
I20260929 14:42:36.390296 8077 raft_consensus.cc:2454] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 0.
I20260929 14:42:36.390324 8356 raft_consensus.cc:2454] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 0.
I20260929 14:42:36.390677 8150 leader_election.cc:302] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: e7c3383dc73d4c35b786f53b97435604, eed413db696e4ee69138448595f5ac64, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:36.390910 8511 raft_consensus.cc:2781] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:36.390959 8511 raft_consensus.cc:482] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:36.390990 8511 raft_consensus.cc:3037] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.392189 8511 raft_consensus.cc:504] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.392956 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35939c614bd0455085a1965d2f01b58e" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed413db696e4ee69138448595f5ac64"
I20260929 14:42:36.392969 8077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35939c614bd0455085a1965d2f01b58e" candidate_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c3383dc73d4c35b786f53b97435604"
I20260929 14:42:36.393082 8356 raft_consensus.cc:3037] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.393093 8077 raft_consensus.cc:3037] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:36.394254 8356 raft_consensus.cc:2454] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 1.
I20260929 14:42:36.394361 8077 raft_consensus.cc:2454] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4d8c90759a4c8eac8d4fbb7834d988 in term 1.
I20260929 14:42:36.394589 8150 leader_election.cc:302] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eed413db696e4ee69138448595f5ac64, fb4d8c90759a4c8eac8d4fbb7834d988; no voters:
I20260929 14:42:36.394775 8439 raft_consensus.cc:2781] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:36.394850 8439 raft_consensus.cc:686] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 LEADER]: Becoming Leader. State: Replica: fb4d8c90759a4c8eac8d4fbb7834d988, State: Running, Role: LEADER
I20260929 14:42:36.394984 8439 consensus_queue.cc:237] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } }
I20260929 14:42:36.395639 8511 leader_election.cc:288] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [CANDIDATE]: Term 1 election: Requested vote from peers e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931), eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987)
I20260929 14:42:36.396678 7920 catalog_manager.cc:5495] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4d8c90759a4c8eac8d4fbb7834d988" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 } health_report { overall_health: UNKNOWN } } }
I20260929 14:42:36.434617 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.431099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50955 (local address 127.0.98.3:33987)
0929 14:42:36.431207 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:36.431211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:36.431346 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:42:36.431494 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:36.431498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:36.431546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:36.431617 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:36.431908 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.432527 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.433535 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.433766 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.433856 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.434132 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:36.434299 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:36.434371 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:36.434449 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 14:42:36.434496 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:42:36.434628 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.431001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33987 (local address 127.0.98.2:50955)
0929 14:42:36.431273 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:36.431286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:36.431375 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:36.431668 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:36.431671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:36.431678 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:36.431779 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.431784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.432636 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.432639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.433421 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.433432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.433874 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.433877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.433943 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.433987 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:36.434383 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:36.434389 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:36.434433 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 14:42:36.434507 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260929 14:42:36.441289 8356 raft_consensus.cc:1261] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Refusing update from remote peer fb4d8c90759a4c8eac8d4fbb7834d988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:36.441370 8077 raft_consensus.cc:1261] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Refusing update from remote peer fb4d8c90759a4c8eac8d4fbb7834d988: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:36.441740 8439 consensus_queue.cc:1035] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c3383dc73d4c35b786f53b97435604" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:36.441903 8511 consensus_queue.cc:1035] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed413db696e4ee69138448595f5ac64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:36.457660 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.454235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51913 (local address 127.0.98.2:34695)
0929 14:42:36.454341 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:36.454344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:36.454538 (+ 194us) server_negotiation.cc:407] Connection header received
0929 14:42:36.454647 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:36.454650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:36.454708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:36.454783 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:36.455006 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.455521 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.456538 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.456840 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.456975 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.457213 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:36.457301 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:36.457364 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:36.457462 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 14:42:36.457519 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 14:42:36.457662 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.454182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34695 (local address 127.0.98.3:51913)
0929 14:42:36.454441 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:36.454457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:36.454562 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:36.454804 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:36.454807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:36.454815 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 14:42:36.454923 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.454927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.455636 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.455639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.456419 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:36.456429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:36.456977 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:36.456981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:36.457067 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:36.457097 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 14:42:36.457387 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 14:42:36.457394 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:42:36.457452 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:42:36.457536 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260929 14:42:36.478374 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:36.478523 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:36.480556 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bdf1a1f96ae4fb2a92e5de5d77f10de: Sending DeleteTablet for 3 replicas of tablet 35939c614bd0455085a1965d2f01b58e
I20260929 14:42:36.481236 8337 tablet_service.cc:1514] Processing DeleteTablet for tablet 35939c614bd0455085a1965d2f01b58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46006
I20260929 14:42:36.481218 8197 tablet_service.cc:1514] Processing DeleteTablet for tablet 35939c614bd0455085a1965d2f01b58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33356
I20260929 14:42:36.481220 8057 tablet_service.cc:1514] Processing DeleteTablet for tablet 35939c614bd0455085a1965d2f01b58e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43926
I20260929 14:42:36.481405 8454 tablet_replica.cc:330] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: stopping tablet replica
I20260929 14:42:36.481405 8455 tablet_replica.cc:330] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: stopping tablet replica
I20260929 14:42:36.481510 8454 raft_consensus.cc:2229] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:36.481518 8455 raft_consensus.cc:2229] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:36.481537 8453 tablet_replica.cc:330] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604: stopping tablet replica
I20260929 14:42:36.481617 8455 raft_consensus.cc:2258] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:36.481652 8453 raft_consensus.cc:2229] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:36.481714 8454 raft_consensus.cc:2258] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:36.481765 8453 raft_consensus.cc:2258] T 35939c614bd0455085a1965d2f01b58e P e7c3383dc73d4c35b786f53b97435604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:36.482065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7992
I20260929 14:42:36.482512 8454 ts_tablet_manager.cc:1905] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:36.483151 8455 ts_tablet_manager.cc:1905] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:36.485026 8454 ts_tablet_manager.cc:1918] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:36.485110 8454 log.cc:1196] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/35939c614bd0455085a1965d2f01b58e
I20260929 14:42:36.485321 8454 ts_tablet_manager.cc:1939] T 35939c614bd0455085a1965d2f01b58e P fb4d8c90759a4c8eac8d4fbb7834d988: Deleting consensus metadata
I20260929 14:42:36.485581 8455 ts_tablet_manager.cc:1918] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 14:42:36.485658 8455 log.cc:1196] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/35939c614bd0455085a1965d2f01b58e
I20260929 14:42:36.485860 8455 ts_tablet_manager.cc:1939] T 35939c614bd0455085a1965d2f01b58e P eed413db696e4ee69138448595f5ac64: Deleting consensus metadata
I20260929 14:42:36.485949 7906 catalog_manager.cc:4841] TS fb4d8c90759a4c8eac8d4fbb7834d988 (127.0.98.2:34695): tablet 35939c614bd0455085a1965d2f01b58e (table test-table [id=94b1e36949ca43c588b11e0918333e96]) successfully deleted
I20260929 14:42:36.486524 7906 catalog_manager.cc:4841] TS eed413db696e4ee69138448595f5ac64 (127.0.98.3:33987): tablet 35939c614bd0455085a1965d2f01b58e (table test-table [id=94b1e36949ca43c588b11e0918333e96]) successfully deleted
W20260929 14:42:36.490196 7905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32931 (error 108)
I20260929 14:42:36.490434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8132
I20260929 14:42:36.492277 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:36.491879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32931 (local address 127.0.0.1:43950)
0929 14:42:36.492097 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:36.492174 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":57,"threads_started":1}
W20260929 14:42:36.492450 7905 catalog_manager.cc:4584] TS e7c3383dc73d4c35b786f53b97435604 (127.0.98.1:32931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35939c614bd0455085a1965d2f01b58e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32931: connect: Connection refused (error 111)
I20260929 14:42:36.496745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8272
I20260929 14:42:36.502648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 7888
2026-09-29T14:42:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3108 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:36 dist-test-slave-g8hp krb5kdc[8531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:36 dist-test-slave-g8hp krb5kdc[8531](info): set up 2 sockets
Sep 29 14:42:36 dist-test-slave-g8hp krb5kdc[8531](info): commencing operation
krb5kdc: starting...
W20260929 14:42:37.477349 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:37 dist-test-slave-g8hp krb5kdc[8531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:37.604028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:37.714475 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:37.714772 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:37.714829 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:37.718879 8544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:37.718954 8544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:37.718981 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:37.719002 8544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:37.719022 8544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:37.724426 8544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:37.725625 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:37.726672 8544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:37.735667 8552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:37.735679 8549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:37.735724 8544 server_base.cc:1034] running on GCE node
W20260929 14:42:37.735672 8550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:37.736567 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:37.737778 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:37.738956 8544 hybrid_clock.cc:648] HybridClock initialized: now 1790692957738930 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T14:42:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1239 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:37 dist-test-slave-g8hp krb5kdc[8557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:37 dist-test-slave-g8hp krb5kdc[8557](info): set up 2 sockets
Sep 29 14:42:37 dist-test-slave-g8hp krb5kdc[8557](info): commencing operation
krb5kdc: starting...
W20260929 14:42:38.705480 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:38 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:38.825896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:38.939076 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:38.939409 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:38.939471 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:38.943816 8570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:38.943917 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:38.943943 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:38.943964 8570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:38.943984 8570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:38.949756 8570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:38.950994 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:38.952133 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:38.961156 8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:38.961133 8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:38.961133 8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:38.961477 8570 server_base.cc:1034] running on GCE node
I20260929 14:42:38.962174 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:38.963555 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:38.964747 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790692958964724 us; error 38 us; skew 500 ppm
Sep 29 14:42:38 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:38.974481 8570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:38.975636 8570 webserver.cc:460] Webserver started at http://127.0.98.62:45523/ using document root <none> and password file <none>
I20260929 14:42:38.976240 8570 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:38.976316 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:38.976514 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:38.978734 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "cfcf9b928f7e426a90d0b5bef144b585"
format_stamp: "Formatted at 2026-09-29 14:42:38 on dist-test-slave-g8hp"
server_key: "7b477c28f015b6bce63e981317cc958e"
server_key_iv: "9e3f4730f892ce9239b6fef228c46e20"
server_key_version: "encryptionkey@0"
I20260929 14:42:38.979332 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "cfcf9b928f7e426a90d0b5bef144b585"
format_stamp: "Formatted at 2026-09-29 14:42:38 on dist-test-slave-g8hp"
server_key: "7b477c28f015b6bce63e981317cc958e"
server_key_iv: "9e3f4730f892ce9239b6fef228c46e20"
server_key_version: "encryptionkey@0"
I20260929 14:42:38.983371 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 14:42:38.986183 8585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:38.987741 8570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:38.987881 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "cfcf9b928f7e426a90d0b5bef144b585"
format_stamp: "Formatted at 2026-09-29 14:42:38 on dist-test-slave-g8hp"
server_key: "7b477c28f015b6bce63e981317cc958e"
server_key_iv: "9e3f4730f892ce9239b6fef228c46e20"
server_key_version: "encryptionkey@0"
I20260929 14:42:38.988021 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:39.004460 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:39.008049 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:39.008249 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:39.015189 8570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45099
I20260929 14:42:39.015213 8637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45099 every 8 connection(s)
I20260929 14:42:39.016184 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:39.019021 8638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:39.022832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8570
I20260929 14:42:39.022976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:39.023226 392 external_mini_cluster.cc:1428] Setting key 516d5602da3f9c96cc14b2393de6bfa4
I20260929 14:42:39.025240 8638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Bootstrap starting.
I20260929 14:42:39.027812 8638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:39.028580 8638 log.cc:824] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:39.030808 8638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: No bootstrap required, opened a new log
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:39.034879 8638 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:39.035210 8638 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:39.035279 8638 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcf9b928f7e426a90d0b5bef144b585, State: Initialized, Role: FOLLOWER
I20260929 14:42:39.035782 8638 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:39.035923 8638 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:39.036000 8638 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:39.036098 8638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:39.037087 8638 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:39.037458 8638 leader_election.cc:302] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfcf9b928f7e426a90d0b5bef144b585; no voters:
I20260929 14:42:39.037781 8638 leader_election.cc:288] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:39.037879 8643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:39.038085 8643 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 LEADER]: Becoming Leader. State: Replica: cfcf9b928f7e426a90d0b5bef144b585, State: Running, Role: LEADER
I20260929 14:42:39.038130 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.024665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.98.62:45099)
0929 14:42:39.025107 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:39.025122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:39.025158 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:39.025818 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:39.025835 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:39.026198 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:39.026487 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:39.027383 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.028486 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.029551 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.030044 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.030167 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.033024 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:39.033031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:39.033051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:39.033082 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:39.034851 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.035348 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.035354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.035361 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.035517 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.035799 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.035802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.035804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.036322 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:39.036523 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:39.036804 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 14:42:39.037172 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":136,"threads_started":1}
I20260929 14:42:39.038486 8643 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:39.039202 8638 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:39.040211 8645 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfcf9b928f7e426a90d0b5bef144b585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260929 14:42:39.040429 8645 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:39.040798 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfcf9b928f7e426a90d0b5bef144b585. Latest consensus state: current_term: 1 leader_uuid: "cfcf9b928f7e426a90d0b5bef144b585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260929 14:42:39.040992 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: This master's current role is: LEADER
W20260929 14:42:39.043852 8658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:42:39.043926 8658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:42:39.044034 8652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:39.045033 8652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:39.051040 8652 catalog_manager.cc:1347] Generated new cluster ID: ecf7906f77b94ee4b6298d285b1318b0
I20260929 14:42:39.051141 8652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:39.065138 8652 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:39.066483 8652 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:39.074812 8652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Generated new TSK 0
I20260929 14:42:39.075608 8652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:39.113678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:39.227218 8664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:39.227505 8664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:39.227548 8664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:39.231683 8664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:39.231757 8664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:39.231820 8664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:39.237362 8664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:39.238469 8664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:39.239496 8664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:39.249099 8672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:39.249104 8670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:39.249104 8669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:39.249387 8664 server_base.cc:1034] running on GCE node
I20260929 14:42:39.249837 8664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:39.250490 8664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:39.251659 8664 hybrid_clock.cc:648] HybridClock initialized: now 1790692959251646 us; error 29 us; skew 500 ppm
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:39.254384 8664 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:39.255219 8664 webserver.cc:460] Webserver started at http://127.0.98.1:44159/ using document root <none> and password file <none>
I20260929 14:42:39.255722 8664 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:39.255779 8664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:39.255941 8664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:39.258023 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "dee41ec0b0dc470e9a2bc468a9cf10d3"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "673e80fb433dc451255ff6406e5fbdc9"
server_key_iv: "7924d64290564f18c3097922e515e50d"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.258562 8664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dee41ec0b0dc470e9a2bc468a9cf10d3"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "673e80fb433dc451255ff6406e5fbdc9"
server_key_iv: "7924d64290564f18c3097922e515e50d"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.262385 8664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:39.264838 8679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.266166 8664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:42:39.266290 8664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "dee41ec0b0dc470e9a2bc468a9cf10d3"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "673e80fb433dc451255ff6406e5fbdc9"
server_key_iv: "7924d64290564f18c3097922e515e50d"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.266404 8664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:39.283895 8664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:39.287128 8664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:39.287312 8664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:39.287895 8664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:39.288720 8664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:39.288774 8664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.288817 8664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:39.288846 8664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.297574 8664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45233
I20260929 14:42:39.297596 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45233 every 8 connection(s)
I20260929 14:42:39.298475 8664 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:39.300915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8664
I20260929 14:42:39.301043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:39.301366 392 external_mini_cluster.cc:1428] Setting key 4d14aad16917ee7b0f75dc6a447597e3
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:39.313524 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.300339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60623 (local address 127.0.98.62:45099)
0929 14:42:39.300471 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:39.300475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:39.301442 (+ 967us) server_negotiation.cc:407] Connection header received
0929 14:42:39.302961 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:39.302964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:39.303006 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:39.303077 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:39.304620 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.305209 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.306730 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.307022 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.307116 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.310024 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:39.310027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:39.310029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:39.310054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:39.311512 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.312018 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.312021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.312022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.312098 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.312415 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.312418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.312420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.312604 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:39.312688 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:39.313226 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 14:42:39.313355 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:42:39.314528 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.300718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:60623)
0929 14:42:39.301262 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:39.301324 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:39.302729 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:39.303277 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:39.303286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:39.303760 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:39.304436 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.304456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.305328 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.305331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.306591 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.306603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.307190 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.307194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.307407 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.308050 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:39.308073 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:39.309821 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:39.311621 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.311626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.311640 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.311921 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.312183 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.312186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.312188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.312328 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.312703 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:39.312709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:39.312999 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 14:42:39.313169 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:42:39.313543 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":144,"threads_started":1}
I20260929 14:42:39.315812 8793 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260929 14:42:39.316087 8793 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:39.316659 8793 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260929 14:42:39.318380 8602 ts_manager.cc:194] Registered new tserver with Master: dee41ec0b0dc470e9a2bc468a9cf10d3 (127.0.98.1:45233)
I20260929 14:42:39.319662 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:39.333129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:39.442976 8798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:39.443265 8798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:39.443325 8798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:39.447451 8798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:39.447520 8798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:39.447602 8798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:39.453020 8798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:39.454079 8798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:39.455087 8798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:39.464685 8806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:39.464738 8798 server_base.cc:1034] running on GCE node
W20260929 14:42:39.464686 8803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:39.464685 8804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:39.465344 8798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:39.466032 8798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:39.467214 8798 hybrid_clock.cc:648] HybridClock initialized: now 1790692959467197 us; error 41 us; skew 500 ppm
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:39.470067 8798 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:39.471017 8798 webserver.cc:460] Webserver started at http://127.0.98.2:34805/ using document root <none> and password file <none>
I20260929 14:42:39.471566 8798 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:39.471632 8798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:39.471822 8798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:39.473953 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "82b1f69469d74351998f90aeaa110cef"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "f8912290d68b0350dcaef9f5890d3445"
server_key_iv: "adf60b6a7108219217ccf5a2d32bbf27"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.474498 8798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "82b1f69469d74351998f90aeaa110cef"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "f8912290d68b0350dcaef9f5890d3445"
server_key_iv: "adf60b6a7108219217ccf5a2d32bbf27"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.478246 8798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:39.480975 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.482287 8798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:39.482416 8798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "82b1f69469d74351998f90aeaa110cef"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "f8912290d68b0350dcaef9f5890d3445"
server_key_iv: "adf60b6a7108219217ccf5a2d32bbf27"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.482525 8798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:39.496685 8798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:39.499930 8798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:39.500115 8798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:39.500708 8798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:39.501544 8798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:39.501590 8798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.501642 8798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:39.501670 8798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.509980 8798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40979
I20260929 14:42:39.510005 8926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40979 every 8 connection(s)
I20260929 14:42:39.510864 8798 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:39.519470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8798
I20260929 14:42:39.519590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:39.519858 392 external_mini_cluster.cc:1428] Setting key d2bb08bafca1297af684d3dfa3271e6f
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:39.525867 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.513014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:42653)
0929 14:42:39.513476 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:39.513522 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:39.514775 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:39.515373 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:39.515380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:39.515739 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:39.516229 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.516242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.517039 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.517041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.517924 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.517933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.518407 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.518411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.518584 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.519133 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:39.519148 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:39.520612 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:39.522487 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.522494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.522512 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.522899 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.523160 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.523163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.523165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.523310 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.523818 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:39.523825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:39.524157 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 14:42:39.524359 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 14:42:39.524713 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260929 14:42:39.533671 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.512577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42653 (local address 127.0.98.62:45099)
0929 14:42:39.512711 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:39.512715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:39.513630 (+ 915us) server_negotiation.cc:407] Connection header received
0929 14:42:39.515055 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:39.515058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:39.515114 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:39.515187 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:39.516376 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.516929 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.518042 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.518266 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.518361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.520761 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:39.520764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:39.520766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:39.520790 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:39.522369 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.522987 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.522991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.522994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.523076 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.523430 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.523434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.523435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.523664 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:39.523736 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:39.524522 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 14:42:39.533483 (+ 8961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:42:39.535259 8927 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260929 14:42:39.535558 8927 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:39.536234 8927 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260929 14:42:39.537647 8602 ts_manager.cc:194] Registered new tserver with Master: 82b1f69469d74351998f90aeaa110cef (127.0.98.2:40979)
I20260929 14:42:39.538440 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42653
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:39.564630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:39.676139 8932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:39.676450 8932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:39.676496 8932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:39.680765 8932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:39.680846 8932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:39.680963 8932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:39.686702 8932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:39.687950 8932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:39.689057 8932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:39.699378 8937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:39.699391 8940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:39.699440 8938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:39.699566 8932 server_base.cc:1034] running on GCE node
I20260929 14:42:39.700199 8932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:39.700836 8932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:39.702054 8932 hybrid_clock.cc:648] HybridClock initialized: now 1790692959702033 us; error 38 us; skew 500 ppm
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:39.704977 8932 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:39.705878 8932 webserver.cc:460] Webserver started at http://127.0.98.3:39349/ using document root <none> and password file <none>
I20260929 14:42:39.706487 8932 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:39.706558 8932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:39.706760 8932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:39.708828 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "22bda1d5fa9546148744f2cfc75b9a91"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "2a42c04e2d54e37ee97611f1667e9eb1"
server_key_iv: "6260d3e4eba885908e111b34c5b9d70f"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.709414 8932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22bda1d5fa9546148744f2cfc75b9a91"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "2a42c04e2d54e37ee97611f1667e9eb1"
server_key_iv: "6260d3e4eba885908e111b34c5b9d70f"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.713320 8932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:39.715781 8947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.717242 8932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:39.717358 8932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "22bda1d5fa9546148744f2cfc75b9a91"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "2a42c04e2d54e37ee97611f1667e9eb1"
server_key_iv: "6260d3e4eba885908e111b34c5b9d70f"
server_key_version: "encryptionkey@0"
I20260929 14:42:39.717468 8932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:39.735991 8932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:39.739423 8932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:39.739645 8932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:39.740276 8932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:39.741226 8932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:39.741277 8932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.741333 8932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:39.741361 8932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:39.750211 8932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41217
I20260929 14:42:39.750281 9060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41217 every 8 connection(s)
I20260929 14:42:39.751121 8932 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 14:42:39 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:39.761229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 8932
I20260929 14:42:39.761353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:39.761616 392 external_mini_cluster.cc:1428] Setting key 0068ea64077ec954c35c3bdb4c54b49b
I20260929 14:42:39.765735 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.753149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47375 (local address 127.0.98.62:45099)
0929 14:42:39.753294 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:39.753298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:39.754173 (+ 875us) server_negotiation.cc:407] Connection header received
0929 14:42:39.755593 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:39.755597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:39.755659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:39.755730 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:39.756951 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.757478 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.758731 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.758962 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.759051 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.761710 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:39.761714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:39.761716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:39.761747 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:39.763231 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.763801 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.763804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.763806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.763914 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:39.764387 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:39.764390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:39.764393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:39.764623 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:39.764712 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:39.765484 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 14:42:39.765596 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:42:39.766557 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:39.753549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:47375)
0929 14:42:39.754012 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:39.754057 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:39.755353 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:39.755924 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:39.755931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:39.756296 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:39.756794 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.756809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.757605 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.757608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.758618 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:39.758629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:39.759108 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:39.759112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:39.759272 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:39.759852 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:39.759868 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:39.761526 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:39.763350 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.763356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.763370 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.763677 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.764048 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:39.764052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:39.764054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:39.764252 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:39.764732 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:39.764738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:39.765216 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 14:42:39.765412 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 14:42:39.765711 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260929 14:42:39.768028 9061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260929 14:42:39.768311 9061 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:39.768854 9061 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260929 14:42:39.770062 8602 ts_manager.cc:194] Registered new tserver with Master: 22bda1d5fa9546148744f2cfc75b9a91 (127.0.98.3:41217)
I20260929 14:42:39.770686 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47375
I20260929 14:42:39.775774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 14:42:39.852411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8664
I20260929 14:42:39.857957 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8798
I20260929 14:42:39.863098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8932
I20260929 14:42:39.867884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 8570
I20260929 14:42:39.873513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--webserver_interface=127.0.98.62
--webserver_port=45523
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:39.983004 9070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:39.983309 9070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:39.983351 9070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:39.987442 9070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:39.987519 9070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:39.987537 9070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:39.987553 9070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:39.987567 9070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:39.993161 9070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:39.994246 9070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:39.995245 9070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:40.004035 9078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.004037 9076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.004176 9075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.004371 9070 server_base.cc:1034] running on GCE node
I20260929 14:42:40.005115 9070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:40.006302 9070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:40.007481 9070 hybrid_clock.cc:648] HybridClock initialized: now 1790692960007455 us; error 40 us; skew 500 ppm
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:40.010493 9070 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:40.011474 9070 webserver.cc:460] Webserver started at http://127.0.98.62:45523/ using document root <none> and password file <none>
I20260929 14:42:40.012029 9070 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:40.012112 9070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:40.016215 9070 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:40.018702 9085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.020154 9070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:40.020284 9070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "cfcf9b928f7e426a90d0b5bef144b585"
format_stamp: "Formatted at 2026-09-29 14:42:38 on dist-test-slave-g8hp"
server_key: "7b477c28f015b6bce63e981317cc958e"
server_key_iv: "9e3f4730f892ce9239b6fef228c46e20"
server_key_version: "encryptionkey@0"
I20260929 14:42:40.020887 9070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:40.040350 9070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:40.043776 9070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:40.043967 9070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:40.051177 9070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45099
I20260929 14:42:40.051182 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45099 every 8 connection(s)
I20260929 14:42:40.052333 9070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:40.056828 9138 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:42:40.058609 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Bootstrap starting.
I20260929 14:42:40.060051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9070
I20260929 14:42:40.060542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45233
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45099
--webserver_port=44159
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:42:40.075605 9138 log.cc:824] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:40.083262 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:42:40.083674 9138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Bootstrap complete.
I20260929 14:42:40.087846 9138 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:40.088282 9138 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcf9b928f7e426a90d0b5bef144b585, State: Initialized, Role: FOLLOWER
I20260929 14:42:40.088775 9138 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:40.088929 9138 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:40.089001 9138 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:40.089094 9138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:42:40.091140 9138 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:40.091475 9138 leader_election.cc:302] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfcf9b928f7e426a90d0b5bef144b585; no voters:
I20260929 14:42:40.091789 9138 leader_election.cc:288] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:42:40.091895 9142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:42:40.092084 9142 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [term 2 LEADER]: Becoming Leader. State: Replica: cfcf9b928f7e426a90d0b5bef144b585, State: Running, Role: LEADER
I20260929 14:42:40.092377 9142 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260929 14:42:40.093135 9138 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:40.094009 9144 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfcf9b928f7e426a90d0b5bef144b585. Latest consensus state: current_term: 2 leader_uuid: "cfcf9b928f7e426a90d0b5bef144b585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260929 14:42:40.094144 9144 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:40.094429 9143 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cfcf9b928f7e426a90d0b5bef144b585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcf9b928f7e426a90d0b5bef144b585" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260929 14:42:40.094502 9143 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:40.094628 9151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:40.099210 9151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:40.100003 9151 catalog_manager.cc:1259] Loaded cluster ID: ecf7906f77b94ee4b6298d285b1318b0
I20260929 14:42:40.100147 9151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:40.102727 9151 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:40.103546 9151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cfcf9b928f7e426a90d0b5bef144b585: Loaded TSK: 0
I20260929 14:42:40.105099 9151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:42:40.176023 9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:40.176298 9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:40.176342 9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:40.180411 9140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:40.180480 9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:40.180539 9140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:40.186069 9140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44159
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:40.187403 9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:40.188432 9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:40.198494 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.198494 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.198511 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.198943 9140 server_base.cc:1034] running on GCE node
I20260929 14:42:40.199440 9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:40.200250 9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:40.201433 9140 hybrid_clock.cc:648] HybridClock initialized: now 1790692960201409 us; error 39 us; skew 500 ppm
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:40.204536 9140 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 14:42:40.205752 9140 webserver.cc:460] Webserver started at http://127.0.98.1:44159/ using document root <none> and password file <none>
I20260929 14:42:40.206529 9140 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:40.206642 9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:40.212031 9140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 14:42:40.214751 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.216193 9140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:40.216312 9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "dee41ec0b0dc470e9a2bc468a9cf10d3"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "673e80fb433dc451255ff6406e5fbdc9"
server_key_iv: "7924d64290564f18c3097922e515e50d"
server_key_version: "encryptionkey@0"
I20260929 14:42:40.217027 9140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:40.249540 9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:40.252710 9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:40.252898 9140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:40.253494 9140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:40.254307 9140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:40.254380 9140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.254426 9140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:40.254449 9140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.263552 9140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45233
I20260929 14:42:40.263597 9287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45233 every 8 connection(s)
I20260929 14:42:40.264606 9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:40.267642 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9140
I20260929 14:42:40.268196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45099
--webserver_port=34805
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:42:40.278867 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.266795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60689 (local address 127.0.98.62:45099)
0929 14:42:40.267243 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.267259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.267643 (+ 384us) server_negotiation.cc:407] Connection header received
0929 14:42:40.269748 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.269760 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.270121 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.270461 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.271711 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.272727 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.273940 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.274451 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.274609 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.278103 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60689: BlockingRecv error: recv got EOF from 127.0.98.1:60689 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":92,"threads_started":1}
I20260929 14:42:40.278877 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.266912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:60689)
0929 14:42:40.267420 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.267473 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.269261 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.270582 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.270591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.271031 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.271528 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.271543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.272906 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.272910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.273812 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.273822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.274620 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.274624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.274813 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.275478 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.275495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.277851 (+ 2356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":128,"threads_started":1}
W20260929 14:42:40.279374 9288 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45099 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45099: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.285552 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.279858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:40693)
0929 14:42:40.279969 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.279984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.280064 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.280379 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.280382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.280527 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.280625 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.280631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.281742 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.281745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.282317 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.282327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.282960 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.282964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.283045 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.283615 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.283629 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.285410 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:42:40.285907 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.279936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40693 (local address 127.0.98.62:45099)
0929 14:42:40.280071 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.280076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.280115 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:42:40.280158 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.280161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.280210 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.280287 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.280747 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.281590 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.282427 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.282790 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.282896 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.285570 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40693: BlockingRecv error: recv got EOF from 127.0.98.1:40693 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.291743 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.286070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:35865)
0929 14:42:40.286191 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.286204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.286283 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.286656 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.286659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.286832 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.286924 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.286931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.287634 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.287637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.288396 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.288407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.288856 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.288886 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.288962 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.289541 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.289554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.291603 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 14:42:40.291985 9288 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:42:40.292089 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.286364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35865 (local address 127.0.98.62:45099)
0929 14:42:40.286468 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.286472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.286483 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:40.286514 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.286516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.286550 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.286607 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.287028 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.287540 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.288513 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.288754 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.288832 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.291754 (+ 2922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35865: BlockingRecv error: recv got EOF from 127.0.98.1:35865 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 14:42:40.384058 9292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:40.384372 9292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:40.384438 9292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:40.388742 9292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:40.388830 9292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:40.388935 9292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:40.394608 9292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34805
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:40.395706 9292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:40.396752 9292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:40.406612 9300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.406617 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.407058 9292 server_base.cc:1034] running on GCE node
W20260929 14:42:40.407111 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.407539 9292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:40.408152 9292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:40.409336 9292 hybrid_clock.cc:648] HybridClock initialized: now 1790692960409309 us; error 39 us; skew 500 ppm
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:40.412405 9292 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 14:42:40.413317 9292 webserver.cc:460] Webserver started at http://127.0.98.2:34805/ using document root <none> and password file <none>
I20260929 14:42:40.413875 9292 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:40.413954 9292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:40.417970 9292 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:40.420176 9307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.421594 9292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:40.421705 9292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "82b1f69469d74351998f90aeaa110cef"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "f8912290d68b0350dcaef9f5890d3445"
server_key_iv: "adf60b6a7108219217ccf5a2d32bbf27"
server_key_version: "encryptionkey@0"
I20260929 14:42:40.422307 9292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:40.434902 9292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:40.438164 9292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:40.438349 9292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:40.438939 9292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:40.439744 9292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:40.439800 9292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.439855 9292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:40.439881 9292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.448364 9292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40979
I20260929 14:42:40.448414 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40979 every 8 connection(s)
I20260929 14:42:40.449420 9292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:40.455219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9292
I20260929 14:42:40.455657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41217
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45099
--webserver_port=39349
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:42:40.460399 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.451294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51283 (local address 127.0.98.62:45099)
0929 14:42:40.451437 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.451441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.452250 (+ 809us) server_negotiation.cc:407] Connection header received
0929 14:42:40.453654 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.453657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.453711 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.453788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.455048 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.455591 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.456784 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.457064 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.457153 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.460273 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51283: BlockingRecv error: recv got EOF from 127.0.98.2:51283 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:42:40.461067 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.451655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:51283)
0929 14:42:40.452098 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.452141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.453420 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.453953 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.453961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.454396 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.454881 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.454894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.455710 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.455714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.456648 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.456664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.457207 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.457211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.457386 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.457936 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.457952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.460126 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
W20260929 14:42:40.461635 9421 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45099 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45099: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.466985 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.462082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:56609)
0929 14:42:40.462189 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.462204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.462287 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.462620 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.462623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.462777 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.462864 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.462869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.463598 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.463600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.464134 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.464141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.464546 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.464550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.464632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.465141 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.465152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.466853 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:42:40.467072 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.462154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56609 (local address 127.0.98.62:45099)
0929 14:42:40.462261 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.462265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.462300 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:42:40.462486 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.462488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.462530 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.462592 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.462969 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.463499 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.464238 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.464452 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.464536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.466970 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56609: BlockingRecv error: recv got EOF from 127.0.98.2:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.472256 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.467582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:45371)
0929 14:42:40.467692 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.467706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.467777 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.467998 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.468000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.468137 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.468203 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.468208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.468904 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.468907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.469458 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.469467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.469876 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.469879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.469950 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.470463 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.470474 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.472122 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:42:40.472498 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.467676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45371 (local address 127.0.98.62:45099)
0929 14:42:40.467776 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.467780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.467793 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:40.467866 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.467869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.467908 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.467973 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.468300 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.468782 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.469566 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.469781 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.469866 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.472272 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45371: BlockingRecv error: recv got EOF from 127.0.98.2:45371 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 14:42:40.472651 9421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:42:40.568455 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:40.568737 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:40.568782 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:40.572973 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:40.573055 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:40.573115 9424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:40.578575 9424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39349
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:40.579689 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:40.580819 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:40.590993 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.591022 9424 server_base.cc:1034] running on GCE node
W20260929 14:42:40.591071 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:40.591228 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:40.591739 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:40.592391 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:40.593593 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790692960593573 us; error 36 us; skew 500 ppm
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:40.596895 9424 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 14:42:40.597823 9424 webserver.cc:460] Webserver started at http://127.0.98.3:39349/ using document root <none> and password file <none>
I20260929 14:42:40.598383 9424 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:40.598464 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:40.602691 9424 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:42:40.605193 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.606647 9424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:42:40.606812 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "22bda1d5fa9546148744f2cfc75b9a91"
format_stamp: "Formatted at 2026-09-29 14:42:39 on dist-test-slave-g8hp"
server_key: "2a42c04e2d54e37ee97611f1667e9eb1"
server_key_iv: "6260d3e4eba885908e111b34c5b9d70f"
server_key_version: "encryptionkey@0"
I20260929 14:42:40.607381 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:40.654119 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:40.657509 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:40.657684 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:40.658290 9424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:40.659129 9424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:40.659176 9424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.659228 9424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:40.659256 9424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:40.668249 9424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41217
I20260929 14:42:40.668289 9552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41217 every 8 connection(s)
I20260929 14:42:40.669337 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:40.672978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9424
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 14:42:40.681859 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.671392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40009 (local address 127.0.98.62:45099)
0929 14:42:40.671594 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.671598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.672453 (+ 855us) server_negotiation.cc:407] Connection header received
0929 14:42:40.674146 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.674150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.674202 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.674274 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.675719 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.676426 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.677936 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.678254 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.678472 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.681746 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40009: BlockingRecv error: recv got EOF from 127.0.98.3:40009 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 14:42:40.682519 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.671785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:40009)
0929 14:42:40.672278 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.672327 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.673870 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.674520 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.674532 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.674939 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.675538 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.675557 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.676562 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.676565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.677801 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.677813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.678525 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.678528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.678688 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.679212 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.679229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.681613 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
W20260929 14:42:40.683010 9553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45099 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45099: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260929 14:42:40.683774 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.674055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.0.98.62:45099)
0929 14:42:40.674297 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.674302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.674313 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:42:40.674349 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.674352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.674410 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.674497 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.675257 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.675936 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.676972 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.677311 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.677505 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.679068 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:40.679075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:40.679091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:40.679120 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:40.681047 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:40.681476 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:40.681481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:40.681487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:40.681624 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:40.682105 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:40.682107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:40.682108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:40.682469 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:40.682636 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:40.683359 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 14:42:40.683593 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.688274 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.683454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:48493)
0929 14:42:40.683545 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.683558 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.683624 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.683854 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.683857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.684016 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.684128 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.684135 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.684951 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.684955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.685506 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.685516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.686000 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.686004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.686099 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.686584 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.686597 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.688182 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 14:42:40.688506 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.683520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48493 (local address 127.0.98.62:45099)
0929 14:42:40.683632 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.683635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.683671 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:42:40.683704 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.683706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.683745 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.683825 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.684244 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.684781 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.685616 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.685878 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.685966 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.688310 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48493: BlockingRecv error: recv got EOF from 127.0.98.3:48493 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:40 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:40.693481 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:40.688799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:48811)
0929 14:42:40.688937 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:40.688949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:40.689014 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:40.689219 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:40.689221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:40.689353 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:40.689429 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.689434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.690136 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.690139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.690674 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:40.690682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.691125 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.691127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:40.691187 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.691605 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:40.691617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:40.693285 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:42:40.693583 9290 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:40.688907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48811 (local address 127.0.98.62:45099)
0929 14:42:40.689025 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:40.689029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:40.689041 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:42:40.689070 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:40.689072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:40.689104 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:40.689164 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:40.689516 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.690035 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.690767 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:40.691024 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:40.691100 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:40.693459 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48811: BlockingRecv error: recv got EOF from 127.0.98.3:48811 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 14:42:40.693763 9553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:41.298743 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:41.292611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:33981)
0929 14:42:41.293015 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:41.293032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:41.293168 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:41.293463 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:41.293466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:41.293691 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:41.293851 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.293858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.294742 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.294745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.295418 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.295428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.295926 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.295929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.296014 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.296554 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:41.296570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:41.298571 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":87,"threads_started":1}
W20260929 14:42:41.298938 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:41.292708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33981 (local address 127.0.98.62:45099)
0929 14:42:41.293031 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:41.293037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:41.293108 (+ 71us) server_negotiation.cc:407] Connection header received
0929 14:42:41.293282 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:41.293285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:41.293344 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:41.293440 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:41.293992 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.294577 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.295548 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.295786 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.295881 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.298816 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33981: BlockingRecv error: recv got EOF from 127.0.98.1:33981 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:41.478547 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:41.473278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:57763)
0929 14:42:41.473584 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:41.473599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:41.473711 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:41.474050 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:41.474053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:41.474247 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:41.474371 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.474378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.475101 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.475104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.475674 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.475682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.476096 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.476100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.476190 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.476679 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:41.476692 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:41.478383 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":130,"threads_started":1}
W20260929 14:42:41.478619 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:41.473421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57763 (local address 127.0.98.62:45099)
0929 14:42:41.473565 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:41.473569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:41.473668 (+ 99us) server_negotiation.cc:407] Connection header received
0929 14:42:41.473904 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:41.473906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:41.473951 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:41.474014 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:41.474486 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.475002 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.475786 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.476001 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.476084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.478520 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57763: BlockingRecv error: recv got EOF from 127.0.98.2:57763 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:41 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:41.699947 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:41.694306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:41991)
0929 14:42:41.694654 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:41.694668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:41.694776 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:41.695139 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:41.695141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:41.695307 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:41.695413 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.695418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.696239 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.696241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.696814 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:41.696821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.697314 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.697317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:41.697382 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.697852 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:41.697863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:41.699775 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":69,"threads_started":1}
W20260929 14:42:41.700040 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:41.694359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41991 (local address 127.0.98.62:45099)
0929 14:42:41.694500 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:41.694505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:41.694727 (+ 222us) server_negotiation.cc:407] Connection header received
0929 14:42:41.694950 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:41.694954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:41.695035 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:41.695099 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:41.695557 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.696148 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.696986 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:41.697206 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:41.697284 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:41.699937 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41991: BlockingRecv error: recv got EOF from 127.0.98.3:41991 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:42.305148 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:42.299567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:43671)
0929 14:42:42.299850 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:42.299864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:42.299966 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:42.300191 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:42.300193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:42.300368 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:42.300461 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.300466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.301486 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.301489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.302096 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.302107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.302555 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.302559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.302640 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.303130 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:42.303142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:42.304976 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20260929 14:42:42.305244 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:42.299690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43671 (local address 127.0.98.62:45099)
0929 14:42:42.299940 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:42.299943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:42.299957 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:42.300044 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:42.300047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:42.300089 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:42.300168 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:42.300619 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.301349 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.302218 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.302443 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.302531 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.305118 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43671: BlockingRecv error: recv got EOF from 127.0.98.1:43671 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":102,"threads_started":1}
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:42.485488 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:42.479523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:55281)
0929 14:42:42.479796 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:42.479814 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:42.479914 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:42.480212 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:42.480220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:42.480479 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:42.480577 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.480583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.481595 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.481598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.482425 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.482435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.482887 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.482890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.482960 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.483530 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:42.483542 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:42.485329 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
W20260929 14:42:42.485561 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:42.479666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55281 (local address 127.0.98.62:45099)
0929 14:42:42.479833 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:42.479836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:42.479870 (+ 34us) server_negotiation.cc:407] Connection header received
0929 14:42:42.480035 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:42.480038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:42.480087 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:42.480164 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:42.480716 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.481470 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.482540 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.482760 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.482842 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.485468 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55281: BlockingRecv error: recv got EOF from 127.0.98.2:55281 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:42 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:42.706046 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:42.700772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:48357)
0929 14:42:42.701050 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:42.701064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:42.701151 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:42.701358 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:42.701360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:42.701537 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:42.701630 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.701635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.702391 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.702394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.702955 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:42.702964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.703458 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.703461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:42.703524 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.704071 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:42.704083 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:42.705904 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":114,"threads_started":1}
W20260929 14:42:42.706171 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:42.700970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48357 (local address 127.0.98.62:45099)
0929 14:42:42.701109 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:42.701112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:42.701126 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:42.701223 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:42.701225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:42.701268 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:42.701336 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:42.701714 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.702272 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.703054 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:42.703266 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:42.703335 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:42.706064 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48357: BlockingRecv error: recv got EOF from 127.0.98.3:48357 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:43.311478 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:43.306044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:52757)
0929 14:42:43.306292 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:43.306309 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:43.306396 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:43.306664 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:43.306667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:43.306846 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:43.306954 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.306960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.307752 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.307755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.308337 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.308346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.308783 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.308786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.308940 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.309444 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:43.309456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:43.311334 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
W20260929 14:42:43.311659 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:43.306142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52757 (local address 127.0.98.62:45099)
0929 14:42:43.306394 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:43.306397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:43.306410 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:43.306476 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:43.306478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:43.306519 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:43.306591 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:43.307058 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.307610 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.308455 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.308691 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.308775 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.311499 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52757: BlockingRecv error: recv got EOF from 127.0.98.1:52757 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":71,"threads_started":1}
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:43.491909 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:43.486279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:32779)
0929 14:42:43.486555 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:43.486568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:43.486665 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:43.486941 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:43.486944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:43.487142 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:43.487251 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.487257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.488067 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.488070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.488643 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.488652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.489167 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.489170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.489243 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.489776 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:43.489789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:43.491729 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
W20260929 14:42:43.491981 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:43.486372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32779 (local address 127.0.98.62:45099)
0929 14:42:43.486533 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:43.486537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:43.486630 (+ 93us) server_negotiation.cc:407] Connection header received
0929 14:42:43.486737 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:43.486739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:43.486786 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:43.486861 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:43.487350 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.487935 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.488757 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.489033 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.489115 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.491877 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32779: BlockingRecv error: recv got EOF from 127.0.98.2:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:43 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:43.713003 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:43.707013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:45811)
0929 14:42:43.707274 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:43.707287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:43.707400 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:43.707641 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:43.707643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:43.707815 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:43.707987 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.707994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.708826 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.708830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.709640 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:43.709652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.710217 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.710220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:43.710281 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.710803 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:43.710816 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:43.712821 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
W20260929 14:42:43.713089 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:43.707091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45811 (local address 127.0.98.62:45099)
0929 14:42:43.707277 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:43.707281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:43.707379 (+ 98us) server_negotiation.cc:407] Connection header received
0929 14:42:43.707481 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:43.707483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:43.707531 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:43.707598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:43.708125 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.708706 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.709762 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:43.710108 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:43.710200 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:43.712982 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45811: BlockingRecv error: recv got EOF from 127.0.98.3:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:44.323559 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:44.312228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:59403)
0929 14:42:44.312493 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:44.312506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:44.312595 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:44.312937 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:44.312940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:44.313145 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:44.313244 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.313251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.314119 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.314122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.314734 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.314743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.315181 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.315184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.315252 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.321258 (+ 6006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:44.321276 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:44.323376 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
W20260929 14:42:44.323669 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:44.312330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59403 (local address 127.0.98.62:45099)
0929 14:42:44.312566 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:44.312569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:44.312583 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:44.312730 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:44.312733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:44.312781 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:44.312900 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:44.313461 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.314012 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.314851 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.315075 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.315151 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.323539 (+ 8388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59403: BlockingRecv error: recv got EOF from 127.0.98.1:59403 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:44.498697 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:44.492851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:60049)
0929 14:42:44.493239 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:44.493252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:44.493408 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:44.493744 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:44.493747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:44.493949 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:44.494064 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.494070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.494951 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.494954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.495545 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.495554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.496048 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.496051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.496119 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.496711 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:44.496723 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:44.498522 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":68,"threads_started":1}
W20260929 14:42:44.498888 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:44.492975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60049 (local address 127.0.98.62:45099)
0929 14:42:44.493118 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:44.493122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:44.493439 (+ 317us) server_negotiation.cc:407] Connection header received
0929 14:42:44.493494 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:44.493496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:44.493599 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:44.493686 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:44.494191 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.494842 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.495670 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.495907 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.495994 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.498679 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60049: BlockingRecv error: recv got EOF from 127.0.98.2:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:44 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:44.719997 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:44.713979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:57603)
0929 14:42:44.714247 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:44.714260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:44.714366 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:44.714740 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:44.714744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:44.714978 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:44.715088 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.715095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.716034 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.716037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.716778 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:44.716788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.717309 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.717313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:44.717385 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.717965 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:44.717977 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:44.719864 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
W20260929 14:42:44.720146 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:44.714100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57603 (local address 127.0.98.62:45099)
0929 14:42:44.714276 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:44.714280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:44.714320 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:42:44.714455 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:44.714457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:44.714592 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:44.714677 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:44.715210 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.715915 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.716923 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:44.717165 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:44.717253 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:44.720023 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57603: BlockingRecv error: recv got EOF from 127.0.98.3:57603 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:45.330248 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:45.324435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:46229)
0929 14:42:45.324692 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:45.324710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:45.324836 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:45.325130 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:45.325134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:45.325436 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:45.325548 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.325554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.326428 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.326431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.327058 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.327068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.327481 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.327484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.327559 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.328032 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:45.328043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:45.330108 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
W20260929 14:42:45.330353 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:45.324537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46229 (local address 127.0.98.62:45099)
0929 14:42:45.324751 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:45.324755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:45.324769 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:45.324968 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:45.324971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:45.325019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:45.325093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:45.325671 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.326320 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.327161 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.327381 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.327468 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.330237 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46229: BlockingRecv error: recv got EOF from 127.0.98.1:46229 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:45.505070 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:45.499599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:50865)
0929 14:42:45.499842 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:45.499858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:45.500003 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:45.500266 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:45.500269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:45.500460 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:45.500556 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.500561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.501320 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.501322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.501960 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.501969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.502436 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.502438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.502501 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.502986 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:45.502998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:45.504909 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
W20260929 14:42:45.505223 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:45.499753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50865 (local address 127.0.98.62:45099)
0929 14:42:45.499895 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:45.499900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:45.499920 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:42:45.500100 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:45.500102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:45.500147 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:45.500220 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:45.500657 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.501234 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.502085 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.502319 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.502405 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.505101 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50865: BlockingRecv error: recv got EOF from 127.0.98.2:50865 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:45 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:45.726466 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:45.720967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:51077)
0929 14:42:45.721240 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:45.721257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:45.721366 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:45.721631 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:45.721634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:45.721826 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:45.721919 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.721925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.722725 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.722729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.723347 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:45.723356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.723847 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.723850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:45.723924 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.724426 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:45.724441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:45.726305 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
W20260929 14:42:45.726637 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:45.721162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51077 (local address 127.0.98.62:45099)
0929 14:42:45.721287 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:45.721292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:45.721310 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:42:45.721470 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:45.721473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:45.721528 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:45.721597 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:45.722037 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.722619 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.723479 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:45.723744 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:45.723840 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:45.726476 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51077: BlockingRecv error: recv got EOF from 127.0.98.3:51077 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:46.336972 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:46.331216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:40739)
0929 14:42:46.331527 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:46.331541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:46.331637 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:46.331932 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:46.331935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:46.332114 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:46.332214 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.332220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.333082 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.333085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.333771 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.333783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.334333 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.334336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.334408 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.334936 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:46.334948 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:46.336802 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
W20260929 14:42:46.337129 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:46.331403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40739 (local address 127.0.98.62:45099)
0929 14:42:46.331631 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:46.331634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:46.331649 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:46.331716 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:46.331719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:46.331764 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:46.331853 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:46.332374 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.332949 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.333948 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.334188 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.334288 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.336987 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40739: BlockingRecv error: recv got EOF from 127.0.98.1:40739 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:46.511264 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:46.505902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:38503)
0929 14:42:46.506144 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:46.506158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:46.506253 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:46.506589 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:46.506592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:46.506768 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:46.506869 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.506874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.507602 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.507604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.508150 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.508159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.508588 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.508592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.508671 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.509172 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:46.509183 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:46.511141 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
W20260929 14:42:46.511389 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:46.505941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38503 (local address 127.0.98.62:45099)
0929 14:42:46.506102 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:46.506106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:46.506209 (+ 103us) server_negotiation.cc:407] Connection header received
0929 14:42:46.506348 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:46.506351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:46.506403 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:46.506502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:46.506977 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.507500 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.508255 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.508489 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.508573 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.511277 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38503: BlockingRecv error: recv got EOF from 127.0.98.2:38503 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:46 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:46.733033 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:46.727340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:59527)
0929 14:42:46.727653 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:46.727670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:46.727787 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:46.728047 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:46.728050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:46.728241 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:46.728330 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.728336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.729209 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.729213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.729774 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:46.729784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.730217 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.730220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:46.730301 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.730789 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:46.730800 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:46.732826 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":103,"threads_started":1}
W20260929 14:42:46.733107 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:46.727552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59527 (local address 127.0.98.62:45099)
0929 14:42:46.727711 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:46.727715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:46.727731 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:42:46.727885 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:46.727888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:46.727938 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:46.728006 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:46.728529 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.729104 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.729892 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:46.730111 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:46.730198 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:46.733009 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59527: BlockingRecv error: recv got EOF from 127.0.98.3:59527 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:47.343994 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:47.337985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:40939)
0929 14:42:47.338331 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:47.338344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:47.338445 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:47.338675 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:47.338677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:47.338847 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:47.338940 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.338945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.339687 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.339690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.340268 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.340276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.340836 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.340839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.340968 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.341557 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:47.341573 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:47.343839 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":73,"threads_started":1}
W20260929 14:42:47.344321 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:47.338098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40939 (local address 127.0.98.62:45099)
0929 14:42:47.338329 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:47.338332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:47.338412 (+ 80us) server_negotiation.cc:407] Connection header received
0929 14:42:47.338519 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:47.338522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:47.338565 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:47.338647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:47.339046 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.339581 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.340389 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.340621 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.340700 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.344200 (+ 3500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40939: BlockingRecv error: recv got EOF from 127.0.98.1:40939 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:47.517320 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:47.512120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:54389)
0929 14:42:47.512418 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:47.512438 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:47.512541 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:47.512769 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:47.512772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:47.512998 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:47.513091 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.513097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.513828 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.513831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.514390 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.514398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.514849 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.514853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.514934 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.515417 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:47.515438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:47.517202 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
W20260929 14:42:47.517490 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:47.512174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54389 (local address 127.0.98.62:45099)
0929 14:42:47.512313 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:47.512317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:47.512506 (+ 189us) server_negotiation.cc:407] Connection header received
0929 14:42:47.512625 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:47.512628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:47.512674 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:47.512737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:47.513198 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.513715 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.514503 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.514748 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.514833 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.517364 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54389: BlockingRecv error: recv got EOF from 127.0.98.2:54389 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:47 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:47.739886 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:47.733940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:42831)
0929 14:42:47.734340 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:47.734354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:47.734478 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:47.734769 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:47.734772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:47.734964 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:47.735071 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.735077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.735858 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.735860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.736458 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:47.736468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.736942 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.736945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:47.737009 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.737553 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:47.737568 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:47.739761 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":174,"threads_started":1}
W20260929 14:42:47.740125 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:47.734102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42831 (local address 127.0.98.62:45099)
0929 14:42:47.734259 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:47.734263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:47.734443 (+ 180us) server_negotiation.cc:407] Connection header received
0929 14:42:47.734563 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:47.734566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:47.734616 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:47.734694 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:47.735198 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.735743 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.736591 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:47.736810 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:47.736941 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:47.739915 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42831: BlockingRecv error: recv got EOF from 127.0.98.3:42831 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:48.350517 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:48.344932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:53389)
0929 14:42:48.345180 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:48.345193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:48.345289 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:48.345605 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:48.345608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:48.345795 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:48.345894 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.345900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.346736 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.346739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.347300 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.347308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.347743 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.347746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.347811 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.348330 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:48.348342 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:48.350356 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
W20260929 14:42:48.350692 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:48.345085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53389 (local address 127.0.98.62:45099)
0929 14:42:48.345353 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:48.345357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:48.345370 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:42:48.345408 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:48.345410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:48.345451 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:48.345531 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:48.346009 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.346615 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.347409 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.347636 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.347712 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.350561 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53389: BlockingRecv error: recv got EOF from 127.0.98.1:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:48.524567 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:48.518205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:34793)
0929 14:42:48.518547 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:48.518563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:48.518675 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:48.518921 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:48.518924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:48.519121 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:48.519225 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.519232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.520098 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.520101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.520698 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.520707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.521328 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.521331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.521414 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.522071 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:48.522084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:48.524393 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
W20260929 14:42:48.524670 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:48.518275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34793 (local address 127.0.98.62:45099)
0929 14:42:48.518423 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:48.518428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:48.518642 (+ 214us) server_negotiation.cc:407] Connection header received
0929 14:42:48.518765 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:48.518768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:48.518815 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:48.518880 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:48.519337 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.519993 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.520841 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.521186 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.521314 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.524565 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34793: BlockingRecv error: recv got EOF from 127.0.98.2:34793 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:48 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:48.746438 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:48.740783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:54185)
0929 14:42:48.741092 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:48.741106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:48.741208 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:48.741549 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:48.741551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:48.741722 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:48.741813 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.741818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.742627 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.742629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.743199 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:48.743207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.743738 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.743741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:48.743805 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.744299 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:48.744310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:48.746323 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":114,"threads_started":1}
W20260929 14:42:48.746541 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:48.740954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54185 (local address 127.0.98.62:45099)
0929 14:42:48.741080 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:48.741083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:48.741169 (+ 86us) server_negotiation.cc:407] Connection header received
0929 14:42:48.741319 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:48.741322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:48.741372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:48.741535 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:48.741920 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.742516 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.743313 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:48.743621 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:48.743709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:48.746444 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54185: BlockingRecv error: recv got EOF from 127.0.98.3:54185 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:49.364012 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:49.351479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:58945)
0929 14:42:49.351744 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:49.351758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:49.351850 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:49.352145 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:49.352148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:49.352326 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:49.352438 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.352444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.353263 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.353267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.353927 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.353938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.354427 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.354430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.354495 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.361662 (+ 7167us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:49.361677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:49.363819 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
W20260929 14:42:49.364259 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:49.351625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58945 (local address 127.0.98.62:45099)
0929 14:42:49.351861 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:49.351864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:49.351882 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:42:49.351930 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:49.351933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:49.351978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:49.352064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:49.352561 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.353151 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.354066 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.354308 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.354397 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.364060 (+ 9663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58945: BlockingRecv error: recv got EOF from 127.0.98.1:58945 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:49.532775 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:49.525496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:35561)
0929 14:42:49.525851 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:49.525868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:49.526019 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:49.526719 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:49.526722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:49.526953 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:49.527083 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.527092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.528096 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.528100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.528996 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.529009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.529737 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.529741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.529826 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.530409 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:49.530424 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:49.532618 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":84,"threads_started":1}
W20260929 14:42:49.533226 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:49.525594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35561 (local address 127.0.98.62:45099)
0929 14:42:49.525795 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:49.525799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:49.526304 (+ 505us) server_negotiation.cc:407] Connection header received
0929 14:42:49.526357 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:49.526360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:49.526417 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:49.526492 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:49.527203 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.527950 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.529139 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.529499 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.529593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.533114 (+ 3521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35561: BlockingRecv error: recv got EOF from 127.0.98.2:35561 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:49 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:49.752929 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:49.747446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:36293)
0929 14:42:49.747718 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:49.747731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:49.747835 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:49.748168 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:49.748170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:49.748351 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:49.748444 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.748449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.749376 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.749379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.749940 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:49.749948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.750420 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.750423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:49.750480 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.751001 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:49.751012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:49.752793 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
W20260929 14:42:49.753159 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:49.747551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36293 (local address 127.0.98.62:45099)
0929 14:42:49.747708 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:49.747712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:49.747824 (+ 112us) server_negotiation.cc:407] Connection header received
0929 14:42:49.747924 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:49.747927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:49.748054 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:49.748128 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:49.748553 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.749267 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.750098 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:49.750333 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:49.750410 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:49.752957 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36293: BlockingRecv error: recv got EOF from 127.0.98.3:36293 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:50.370838 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:50.365011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:57987)
0929 14:42:50.365284 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:50.365300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:50.365391 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:50.365612 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:50.365614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:50.365788 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:50.365879 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.365885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.366753 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.366756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.367396 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.367409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.367949 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.367953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.368048 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.368684 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:50.368696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:50.370713 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
W20260929 14:42:50.371068 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:50.365121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57987 (local address 127.0.98.62:45099)
0929 14:42:50.365376 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:50.365379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:50.365394 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:50.365476 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:50.365478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:50.365519 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:50.365584 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:50.366014 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.366619 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.367502 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.367804 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.367914 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.370887 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57987: BlockingRecv error: recv got EOF from 127.0.98.1:57987 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:50.542368 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:50.536780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:41937)
0929 14:42:50.537141 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:50.537156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:50.537253 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:50.537585 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:50.537588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:50.537763 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:50.537865 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.537870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.538662 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.538665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.539264 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.539275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.539787 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.539790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.539860 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.540341 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:50.540352 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:50.542246 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":157,"threads_started":1}
W20260929 14:42:50.542562 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:50.536935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41937 (local address 127.0.98.62:45099)
0929 14:42:50.537121 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:50.537124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:50.537222 (+ 98us) server_negotiation.cc:407] Connection header received
0929 14:42:50.537323 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:50.537326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:50.537376 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:50.537450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:50.537958 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.538470 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.539396 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.539653 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.539737 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.542433 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41937: BlockingRecv error: recv got EOF from 127.0.98.2:41937 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:50 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:50.759378 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:50.753766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:46753)
0929 14:42:50.754091 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:50.754104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:50.754206 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:50.754416 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:50.754418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:50.754598 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:50.754755 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.754761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.755534 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.755537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.756145 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:50.756154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.756676 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.756679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:50.756744 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.757320 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:50.757331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:50.759260 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
W20260929 14:42:50.759590 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:50.753971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46753 (local address 127.0.98.62:45099)
0929 14:42:50.754118 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:50.754122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:50.754161 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:42:50.754282 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:50.754284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:50.754327 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:50.754391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:50.754879 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.755419 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.756255 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:50.756488 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:50.756565 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:50.759445 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46753: BlockingRecv error: recv got EOF from 127.0.98.3:46753 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:51.377645 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:51.371668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:50813)
0929 14:42:51.372091 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:51.372103 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:51.372220 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:51.372510 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:51.372512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:51.372696 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:51.372795 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.372800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.373645 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.373648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.374341 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.374352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.374810 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.374813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.374943 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.375461 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:51.375473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:51.377520 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
W20260929 14:42:51.377833 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:51.371727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50813 (local address 127.0.98.62:45099)
0929 14:42:51.371974 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:51.371979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:51.372177 (+ 198us) server_negotiation.cc:407] Connection header received
0929 14:42:51.372334 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:51.372337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:51.372394 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:51.372483 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:51.372970 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.373533 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.374466 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.374704 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.374787 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.377687 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50813: BlockingRecv error: recv got EOF from 127.0.98.1:50813 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:51.548746 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:51.543204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:41897)
0929 14:42:51.543520 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:51.543535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:51.543651 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:51.543946 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:51.543949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:51.544155 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:51.544251 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.544257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.545173 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.545176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.545750 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.545759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.546213 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.546216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.546298 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.546821 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:51.546835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:51.548616 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
W20260929 14:42:51.548913 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:51.543312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41897 (local address 127.0.98.62:45099)
0929 14:42:51.543486 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:51.543491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:51.543608 (+ 117us) server_negotiation.cc:407] Connection header received
0929 14:42:51.543749 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:51.543782 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:51.543830 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:51.543907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:51.544364 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.545065 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.545862 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.546105 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.546194 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.548775 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41897: BlockingRecv error: recv got EOF from 127.0.98.2:41897 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:51 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:51.765669 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:51.760164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:46095)
0929 14:42:51.760444 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:51.760459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:51.760577 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:51.760877 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:51.760880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:51.761084 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:51.761246 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.761253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.762007 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.762011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.762629 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:51.762639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.763098 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.763102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:51.763188 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.763694 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:51.763709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:51.765513 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260929 14:42:51.765782 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:51.760298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46095 (local address 127.0.98.62:45099)
0929 14:42:51.760444 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:51.760448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:51.760520 (+ 72us) server_negotiation.cc:407] Connection header received
0929 14:42:51.760707 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:51.760709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:51.760753 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:51.760820 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:51.761373 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.761901 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.762740 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:51.762993 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:51.763082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:51.765681 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46095: BlockingRecv error: recv got EOF from 127.0.98.3:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:52.384603 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:52.378463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:51997)
0929 14:42:52.378715 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:52.378730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:52.378831 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:52.379159 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:52.379162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:52.379347 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:52.379543 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.379550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.380537 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.380540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.381165 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.381176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.381726 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.381729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.381799 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.382423 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:52.382439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:52.384442 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
W20260929 14:42:52.384815 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:52.378628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51997 (local address 127.0.98.62:45099)
0929 14:42:52.378887 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:52.378891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:52.378905 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:52.378946 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:52.378949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:52.378992 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:52.379079 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:52.379693 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.380287 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.381282 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.381590 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.381682 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.384695 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51997: BlockingRecv error: recv got EOF from 127.0.98.1:51997 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:52.556371 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:52.549815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:50927)
0929 14:42:52.550155 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:52.550170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:52.550289 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:52.550668 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:52.550671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:52.550850 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:52.550949 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.550954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.551796 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.551799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.552387 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.552396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.553068 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.553077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.553166 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.553879 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:52.553895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:52.556247 (+ 2352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":159,"threads_started":1}
W20260929 14:42:52.556634 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:52.549905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50927 (local address 127.0.98.62:45099)
0929 14:42:52.550094 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:52.550100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:52.550285 (+ 185us) server_negotiation.cc:407] Connection header received
0929 14:42:52.550400 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:52.550402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:52.550453 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:52.550619 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:52.551069 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.551678 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.552609 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.552885 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.552996 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.556434 (+ 3438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50927: BlockingRecv error: recv got EOF from 127.0.98.2:50927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:52 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:52.772626 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:52.766569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:48539)
0929 14:42:52.766900 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:52.766913 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:52.767020 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:52.767318 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:52.767320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:52.767501 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:52.767604 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.767611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.768501 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.768504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.769137 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:52.769149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.769743 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.769746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:52.769816 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.770438 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:52.770452 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:52.772474 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":150,"threads_started":1}
W20260929 14:42:52.772732 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:52.766745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48539 (local address 127.0.98.62:45099)
0929 14:42:52.766910 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:52.766914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:52.766983 (+ 69us) server_negotiation.cc:407] Connection header received
0929 14:42:52.767107 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:52.767110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:52.767166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:52.767240 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:52.767784 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.768355 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.769273 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:52.769595 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:52.769689 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:52.772620 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48539: BlockingRecv error: recv got EOF from 127.0.98.3:48539 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:53.391522 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:53.385627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:50515)
0929 14:42:53.386024 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:53.386037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:53.386146 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:53.386396 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:53.386398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:53.386574 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:53.386672 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.386677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.387572 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.387576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.388229 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.388242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.388753 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.388757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.388827 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.389539 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:53.389551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:53.391376 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":89,"threads_started":1}
W20260929 14:42:53.391708 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:53.385710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50515 (local address 127.0.98.62:45099)
0929 14:42:53.385975 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:53.385978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:53.386111 (+ 133us) server_negotiation.cc:407] Connection header received
0929 14:42:53.386228 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:53.386230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:53.386274 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:53.386352 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:53.386783 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.387448 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.388339 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.388620 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.388715 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.391571 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50515: BlockingRecv error: recv got EOF from 127.0.98.1:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:53.564359 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:53.557425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:50013)
0929 14:42:53.557692 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:53.557708 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:53.557798 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:53.558109 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:53.558112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:53.558303 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:53.558414 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.558420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.559313 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.559317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.560327 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.560341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.561007 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.561011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.561105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.561677 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:53.561692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:53.564174 (+ 2482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
W20260929 14:42:53.564747 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:53.557594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50013 (local address 127.0.98.62:45099)
0929 14:42:53.557741 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:53.557744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:53.557759 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:53.557872 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:53.557874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:53.557916 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:53.558060 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:53.558543 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.559195 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.560457 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.560843 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.560978 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.564632 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50013: BlockingRecv error: recv got EOF from 127.0.98.2:50013 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:53 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:53.779188 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:53.773561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:40401)
0929 14:42:53.773879 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:53.773893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:53.774000 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:53.774384 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:53.774387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:53.774584 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:53.774688 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.774695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.775448 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.775451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.776034 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:53.776043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.776649 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.776652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:53.776723 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.777241 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:53.777253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:53.779033 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
W20260929 14:42:53.779312 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:53.773646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40401 (local address 127.0.98.62:45099)
0929 14:42:53.773789 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:53.773794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:53.774001 (+ 207us) server_negotiation.cc:407] Connection header received
0929 14:42:53.774177 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:53.774179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:53.774274 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:53.774335 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:53.774787 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.775331 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.776260 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:53.776520 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:53.776605 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:53.779200 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40401: BlockingRecv error: recv got EOF from 127.0.98.3:40401 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:54.402572 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:54.392420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:50167)
0929 14:42:54.392740 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:54.392754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:54.392853 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:54.393287 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:54.393290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:54.393522 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:54.393628 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.393634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.394565 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.394569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.395152 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.395161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.395625 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.395628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.395697 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.400439 (+ 4742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:54.400453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:54.402385 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":139,"threads_started":1}
W20260929 14:42:54.402679 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:54.392547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50167 (local address 127.0.98.62:45099)
0929 14:42:54.392936 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:54.392940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:54.392955 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:54.392998 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:54.393001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:54.393052 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:54.393156 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:54.393741 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.394461 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.395248 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.395517 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.395596 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.402554 (+ 6958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50167: BlockingRecv error: recv got EOF from 127.0.98.1:50167 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":70,"threads_started":1}
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:54.570662 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:54.565197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:57811)
0929 14:42:54.565457 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:54.565473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:54.565594 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:54.565851 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:54.565853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:54.566041 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:54.566137 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.566143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.566889 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.566892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.567490 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.567507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.567979 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.567982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.568054 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.568588 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:54.568602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:54.570480 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
W20260929 14:42:54.570744 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:54.565313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57811 (local address 127.0.98.62:45099)
0929 14:42:54.565480 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:54.565484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:54.565572 (+ 88us) server_negotiation.cc:407] Connection header received
0929 14:42:54.565702 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:54.565704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:54.565752 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:54.565828 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:54.566244 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.566783 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.567618 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.567853 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.567943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.570633 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57811: BlockingRecv error: recv got EOF from 127.0.98.2:57811 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:54 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:54.785732 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:54.780042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:50931)
0929 14:42:54.780321 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:54.780338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:54.780458 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:54.780742 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:54.780745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:54.780963 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:54.781086 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.781093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.781999 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.782001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.782575 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:54.782583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.783032 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.783044 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:54.783123 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.783646 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:54.783657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:54.785531 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
W20260929 14:42:54.785957 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:54.780221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50931 (local address 127.0.98.62:45099)
0929 14:42:54.780370 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:54.780375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:54.780396 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:42:54.780565 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:54.780575 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:54.780629 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:54.780700 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:54.781216 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.781872 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.782699 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:54.782924 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:54.783014 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:54.785852 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50931: BlockingRecv error: recv got EOF from 127.0.98.3:50931 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 14:42:55 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:55 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:55.409272 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:55.403753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:41065)
0929 14:42:55.404079 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:55.404097 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:55.404196 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:55.404483 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:55.404485 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:55.404662 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:55.404759 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:55.404764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.405568 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.405571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:55.406207 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:55.406219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.406674 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.406678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:55.406755 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:55.407260 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:55.407272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:55.409149 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":81,"threads_started":1}
W20260929 14:42:55.409416 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:55.403873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41065 (local address 127.0.98.62:45099)
0929 14:42:55.404163 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:55.404166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:55.404181 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:42:55.404316 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:55.404319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:55.404363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:55.404439 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:55.404901 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.405439 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.406336 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.406569 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.406669 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:55.409294 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41065: BlockingRecv error: recv got EOF from 127.0.98.1:41065 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
Sep 29 14:42:55 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 14:42:55 dist-test-slave-g8hp krb5kdc[8557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 14:42:55.577528 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:55.571765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:49877)
0929 14:42:55.572032 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:55.572048 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:55.572169 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:55.572414 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:55.572416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:55.572682 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:55.572795 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:55.572803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.573762 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.573766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:55.574343 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:55.574352 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.574824 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.574827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:55.574899 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:55.575382 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:55.575393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:55.577290 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45099: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
W20260929 14:42:55.577564 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:55.571842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49877 (local address 127.0.98.62:45099)
0929 14:42:55.571968 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:55.571973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:55.572127 (+ 154us) server_negotiation.cc:407] Connection header received
0929 14:42:55.572257 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:55.572260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:55.572305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:55.572385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:55.573023 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.573612 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.574462 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.574698 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.574778 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:55.577459 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49877: BlockingRecv error: recv got EOF from 127.0.98.2:49877 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 14:42:55.679468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9140
I20260929 14:42:55.684603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9292
I20260929 14:42:55.689267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9424
I20260929 14:42:55.693985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9070
2026-09-29T14:42:55Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17951 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T14:42:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:55Z Disabled control of system clock
I20260929 14:42:55.722976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37371
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 14:42:55.834511 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:55.834805 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:55.834838 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:55.839123 9620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:55.839200 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:55.839214 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:55.839237 9620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:55.839252 9620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:55.844240 9620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:55.845427 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:55.846825 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:55.856495 9620 server_base.cc:1034] running on GCE node
W20260929 14:42:55.856426 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:55.856423 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:55.856436 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:55.857385 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:55.858599 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:55.859814 9620 hybrid_clock.cc:648] HybridClock initialized: now 1790692975859760 us; error 38 us; skew 500 ppm
I20260929 14:42:55.861969 9620 webserver.cc:460] Webserver started at http://127.0.98.62:35993/ using document root <none> and password file <none>
I20260929 14:42:55.862560 9620 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:55.862622 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:55.862800 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:55.865038 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "4faba77aca9249a7b0eb33433f0f42f7"
format_stamp: "Formatted at 2026-09-29 14:42:55 on dist-test-slave-g8hp"
server_key: "5547d3436a87493ae0a974554c769abc"
server_key_iv: "1e6293507e32baaedd53b8d8eda392cb"
server_key_version: "encryptionkey@0"
I20260929 14:42:55.865602 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4faba77aca9249a7b0eb33433f0f42f7"
format_stamp: "Formatted at 2026-09-29 14:42:55 on dist-test-slave-g8hp"
server_key: "5547d3436a87493ae0a974554c769abc"
server_key_iv: "1e6293507e32baaedd53b8d8eda392cb"
server_key_version: "encryptionkey@0"
I20260929 14:42:55.869535 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:42:55.872117 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:55.873626 9620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:55.873741 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "4faba77aca9249a7b0eb33433f0f42f7"
format_stamp: "Formatted at 2026-09-29 14:42:55 on dist-test-slave-g8hp"
server_key: "5547d3436a87493ae0a974554c769abc"
server_key_iv: "1e6293507e32baaedd53b8d8eda392cb"
server_key_version: "encryptionkey@0"
I20260929 14:42:55.873852 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:55.890589 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:55.891291 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:55.891450 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:55.898195 9620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37371
I20260929 14:42:55.898226 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37371 every 8 connection(s)
I20260929 14:42:55.899179 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:55.902436 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:55.908432 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7: Bootstrap starting.
I20260929 14:42:55.909508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9620
I20260929 14:42:55.909598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:55.909879 392 external_mini_cluster.cc:1428] Setting key 7f6df96940ad6310ca835e7f665cb096
I20260929 14:42:55.911357 9687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:55.912215 9687 log.cc:824] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:55.914783 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7: No bootstrap required, opened a new log
I20260929 14:42:55.918705 9687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } }
I20260929 14:42:55.919013 9687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:55.919100 9687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4faba77aca9249a7b0eb33433f0f42f7, State: Initialized, Role: FOLLOWER
I20260929 14:42:55.919564 9687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } }
I20260929 14:42:55.919699 9687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:55.919764 9687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:55.919866 9687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:55.919996 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:55.911095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.98.62:37371)
0929 14:42:55.911864 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:55.911878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:55.911918 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:42:55.912706 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:55.912733 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:55.913249 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:55.913649 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:55.914099 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.915576 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.916514 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:55.917082 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:55.917234 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:55.918064 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:55.918070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:55.918085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:55.918505 (+ 420us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:55.918566 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:55.918588 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:55.918683 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:55.918823 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 14:42:55.919121 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":145,"threads_started":1}
I20260929 14:42:55.921048 9687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } }
I20260929 14:42:55.921427 9687 leader_election.cc:302] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4faba77aca9249a7b0eb33433f0f42f7; no voters:
I20260929 14:42:55.921739 9687 leader_election.cc:288] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:55.921862 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:55.922083 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4faba77aca9249a7b0eb33433f0f42f7, State: Running, Role: LEADER
I20260929 14:42:55.922467 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } }
I20260929 14:42:55.923013 9687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:55.924146 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4faba77aca9249a7b0eb33433f0f42f7. Latest consensus state: current_term: 1 leader_uuid: "4faba77aca9249a7b0eb33433f0f42f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } } }
I20260929 14:42:55.924284 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:55.924619 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4faba77aca9249a7b0eb33433f0f42f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4faba77aca9249a7b0eb33433f0f42f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37371 } } }
I20260929 14:42:55.924727 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:55.924856 9701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:55.927764 9701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:55.933594 9701 catalog_manager.cc:1347] Generated new cluster ID: b5a28aec6ace4c0a9ec592ee66b396c5
I20260929 14:42:55.933670 9701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:55.954487 9701 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:55.955783 9701 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:55.963047 9701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4faba77aca9249a7b0eb33433f0f42f7: Generated new TSK 0
I20260929 14:42:55.963821 9701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:55.975095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--builtin_ntp_servers=127.0.98.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 14:42:56.089048 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:56.089336 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:56.089376 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:56.093596 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:56.093668 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:56.093724 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:56.098565 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:56.099673 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:56.101222 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:56.111074 9711 server_base.cc:1034] running on GCE node
W20260929 14:42:56.111024 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.111033 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.111035 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.111707 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:56.112421 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:56.113595 9711 hybrid_clock.cc:648] HybridClock initialized: now 1790692976113576 us; error 35 us; skew 500 ppm
I20260929 14:42:56.115623 9711 webserver.cc:460] Webserver started at http://127.0.98.1:38653/ using document root <none> and password file <none>
I20260929 14:42:56.116205 9711 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:56.116276 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:56.116474 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:56.118633 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce0f43b89ae54ec9ac0e44406720bd9c"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "02d974a4b320d9dc56653ba31229c174"
server_key_iv: "b9ce0958ab23e6401776522ac4340cd7"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.119194 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce0f43b89ae54ec9ac0e44406720bd9c"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "02d974a4b320d9dc56653ba31229c174"
server_key_iv: "b9ce0958ab23e6401776522ac4340cd7"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.123240 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 14:42:56.126111 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.127460 9711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:56.127586 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "ce0f43b89ae54ec9ac0e44406720bd9c"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "02d974a4b320d9dc56653ba31229c174"
server_key_iv: "b9ce0958ab23e6401776522ac4340cd7"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.127696 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:56.140602 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:56.141407 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:56.141592 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:56.142177 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:56.143229 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:56.143278 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.143332 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:56.143360 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.151690 9711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45939
I20260929 14:42:56.151723 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45939 every 8 connection(s)
I20260929 14:42:56.152598 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:56.161899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9711
I20260929 14:42:56.162019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:56.162264 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.154386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59275 (local address 127.0.98.62:37371)
0929 14:42:56.154589 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.154594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.155752 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 14:42:56.157162 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.157165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.157222 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.157298 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.158510 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.159034 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.160312 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.160530 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.160604 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.161754 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:56.161757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:56.161759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:56.161772 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:56.161792 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:56.161800 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:56.161863 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:56.162077 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 14:42:56.162144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:42:56.162305 392 external_mini_cluster.cc:1428] Setting key 28f35e8e990af3f67c4f11893803eb5e
I20260929 14:42:56.163654 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.154841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37371 (local address 127.0.98.1:59275)
0929 14:42:56.155600 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:56.155643 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:56.156939 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:56.157581 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:56.157589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:56.157687 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:56.158214 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.158228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.159163 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.159166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.160183 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.160193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.160685 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.160688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.160842 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.161464 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:56.161466 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:56.161511 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:56.161579 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:56.161878 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:56.161888 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:56.161894 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:56.162490 (+ 596us) client_negotiation.cc:240] Negotiation successful
0929 14:42:56.162752 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260929 14:42:56.164271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--builtin_ntp_servers=127.0.98.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:42:56.164930 9839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37371
I20260929 14:42:56.165179 9839 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:56.165802 9839 heartbeater.cc:511] Master 127.0.98.62:37371 requested a full tablet report, sending...
I20260929 14:42:56.167431 9651 ts_manager.cc:194] Registered new tserver with Master: ce0f43b89ae54ec9ac0e44406720bd9c (127.0.98.1:45939)
I20260929 14:42:56.168550 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59275
W20260929 14:42:56.277792 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:56.278138 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:56.278205 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:56.282490 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:56.282560 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:56.282644 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:56.287529 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:56.288676 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:56.290165 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:56.300256 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.300357 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.300341 9842 server_base.cc:1034] running on GCE node
W20260929 14:42:56.300285 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.301103 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:56.301772 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:56.303010 9842 hybrid_clock.cc:648] HybridClock initialized: now 1790692976302968 us; error 62 us; skew 500 ppm
I20260929 14:42:56.305094 9842 webserver.cc:460] Webserver started at http://127.0.98.2:33023/ using document root <none> and password file <none>
I20260929 14:42:56.305693 9842 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:56.305764 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:56.305972 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:56.308471 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "43afd3a595b24c219a461f3bd1f9f6e7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "b0f08dc3a7cea43fe641eeff90d1a95f"
server_key_iv: "e3361f16022e1600fc6465d0df03c797"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.309139 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43afd3a595b24c219a461f3bd1f9f6e7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "b0f08dc3a7cea43fe641eeff90d1a95f"
server_key_iv: "e3361f16022e1600fc6465d0df03c797"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.313482 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:42:56.316270 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.317981 9842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:56.318120 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "43afd3a595b24c219a461f3bd1f9f6e7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "b0f08dc3a7cea43fe641eeff90d1a95f"
server_key_iv: "e3361f16022e1600fc6465d0df03c797"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.318246 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:56.336637 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:56.337491 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:56.337682 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:56.338300 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:56.339284 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:56.339334 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.339398 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:56.339424 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.348717 9842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46651
I20260929 14:42:56.348743 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46651 every 8 connection(s)
I20260929 14:42:56.349650 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:56.351439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9842
I20260929 14:42:56.351554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:56.351855 392 external_mini_cluster.cc:1428] Setting key 9adaa7e98de48e15cc6bc4d5bafb8375
I20260929 14:42:56.353986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--builtin_ntp_servers=127.0.98.20:42533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 14:42:56.359364 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.351617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34353 (local address 127.0.98.62:37371)
0929 14:42:56.351751 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.351755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.353145 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 14:42:56.354594 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.354598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.354652 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.354725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.355708 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.356243 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.357523 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.357737 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.357826 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.358905 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:56.358908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:56.358910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:56.358923 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:56.358942 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:56.358950 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:56.359007 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:56.359151 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 14:42:56.359215 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 14:42:56.360239 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.352194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37371 (local address 127.0.98.2:34353)
0929 14:42:56.352987 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:56.353053 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:56.354382 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:56.354919 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:56.354925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:56.355026 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:56.355567 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.355581 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.356368 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.356370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.357386 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.357397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.357880 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.357884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.358056 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.358650 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:56.358651 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:56.358693 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:56.358760 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:56.359024 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:56.359034 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:56.359038 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:42:56.359145 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:42:56.359435 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":123,"threads_started":1}
I20260929 14:42:56.361421 9970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37371
I20260929 14:42:56.361665 9970 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:56.362176 9970 heartbeater.cc:511] Master 127.0.98.62:37371 requested a full tablet report, sending...
I20260929 14:42:56.363211 9651 ts_manager.cc:194] Registered new tserver with Master: 43afd3a595b24c219a461f3bd1f9f6e7 (127.0.98.2:46651)
I20260929 14:42:56.363957 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34353
W20260929 14:42:56.466252 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:56.466529 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:56.466566 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:56.470633 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:56.470697 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:56.470752 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:56.475545 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:56.476610 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:56.478080 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:56.487605 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.487605 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.487623 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.487962 9973 server_base.cc:1034] running on GCE node
I20260929 14:42:56.488415 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:56.489146 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:56.490310 9973 hybrid_clock.cc:648] HybridClock initialized: now 1790692976490284 us; error 36 us; skew 500 ppm
I20260929 14:42:56.492202 9973 webserver.cc:460] Webserver started at http://127.0.98.3:39999/ using document root <none> and password file <none>
I20260929 14:42:56.492779 9973 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:56.492847 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:56.493090 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:56.495247 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3982193362d43b4894b9effc67f0f3a"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "e2972ae36da23644552441a594008134"
server_key_iv: "30b168a25a2e1da6b6859631b26f058e"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.495797 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3982193362d43b4894b9effc67f0f3a"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "e2972ae36da23644552441a594008134"
server_key_iv: "30b168a25a2e1da6b6859631b26f058e"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.499704 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:42:56.502334 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.503623 9973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:42:56.503744 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "e3982193362d43b4894b9effc67f0f3a"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "e2972ae36da23644552441a594008134"
server_key_iv: "30b168a25a2e1da6b6859631b26f058e"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.503854 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:56.523034 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:56.523837 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:56.524016 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:56.524645 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:56.525667 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:56.525720 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.525761 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:56.525775 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.534335 9973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33521
I20260929 14:42:56.534392 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33521 every 8 connection(s)
I20260929 14:42:56.535341 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:56.540258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 9973
I20260929 14:42:56.540364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:56.540591 392 external_mini_cluster.cc:1428] Setting key c8bd00c947881c6e7f0e6b8fbe2aab1e
I20260929 14:42:56.544759 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.537252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38235 (local address 127.0.98.62:37371)
0929 14:42:56.537408 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.537411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.538569 (+ 1158us) server_negotiation.cc:407] Connection header received
0929 14:42:56.540021 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.540026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.540084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.540158 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.541193 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.541722 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.542864 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.543081 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.543167 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.544214 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:56.544218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:56.544221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:56.544235 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:56.544256 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:56.544266 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:56.544326 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:56.544477 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 14:42:56.544540 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:42:56.545616 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.537721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37371 (local address 127.0.98.3:38235)
0929 14:42:56.538405 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:56.538459 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:56.539779 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:56.540355 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:56.540362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:56.540476 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:56.541051 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.541067 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.541841 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.541843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.542745 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:56.542754 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:56.543224 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:56.543228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:56.543391 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:56.543943 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:56.543945 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:56.543987 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:56.544056 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:56.544336 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:56.544347 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:56.544353 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:42:56.544466 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 14:42:56.544770 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260929 14:42:56.546711 10101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37371
I20260929 14:42:56.546932 10101 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:56.547433 10101 heartbeater.cc:511] Master 127.0.98.62:37371 requested a full tablet report, sending...
I20260929 14:42:56.548454 9651 ts_manager.cc:194] Registered new tserver with Master: e3982193362d43b4894b9effc67f0f3a (127.0.98.3:33521)
I20260929 14:42:56.549121 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38235
I20260929 14:42:56.554265 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:42:56.557034 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:56.556208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.0.98.62:37371)
0929 14:42:56.556345 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.556348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.556471 (+ 123us) server_negotiation.cc:407] Connection header received
0929 14:42:56.556567 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.556569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.556609 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.556676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.556919 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33600: BlockingRecv error: recv got EOF from 127.0.0.1:33600 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:56.557478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9711
I20260929 14:42:56.562197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9842
I20260929 14:42:56.566218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9973
I20260929 14:42:56.571005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 9620
2026-09-29T14:42:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (875 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T14:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:56Z Disabled control of system clock
I20260929 14:42:56.597919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43973
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:42:56.707222 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:56.707511 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:56.707551 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:56.711615 10112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:56.711697 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:56.711710 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:56.711726 10112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:56.711740 10112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:56.716658 10112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:56.717954 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:56.719466 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:56.727932 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.728055 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.728235 10112 server_base.cc:1034] running on GCE node
W20260929 14:42:56.728377 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.729055 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:56.730193 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:56.731374 10112 hybrid_clock.cc:648] HybridClock initialized: now 1790692976731339 us; error 44 us; skew 500 ppm
I20260929 14:42:56.733373 10112 webserver.cc:460] Webserver started at http://127.0.98.62:34815/ using document root <none> and password file <none>
I20260929 14:42:56.733938 10112 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:56.734005 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:56.734210 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:56.736260 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "5efcd93614284c14a30da2abdaa5d9f0"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "4bd85cb8b0053f8a8753662902042901"
server_key_iv: "543f6475c4594f44374142697094d734"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.736806 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "5efcd93614284c14a30da2abdaa5d9f0"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "4bd85cb8b0053f8a8753662902042901"
server_key_iv: "543f6475c4594f44374142697094d734"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.740598 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:42:56.743214 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.744490 10112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:42:56.744633 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "5efcd93614284c14a30da2abdaa5d9f0"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "4bd85cb8b0053f8a8753662902042901"
server_key_iv: "543f6475c4594f44374142697094d734"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.744812 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:56.749442 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:56.750059 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:56.750217 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:56.756582 10112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43973
I20260929 14:42:56.756604 10178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43973 every 8 connection(s)
I20260929 14:42:56.757553 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:56.760752 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:56.763489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10112
I20260929 14:42:56.763608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:56.763909 392 external_mini_cluster.cc:1428] Setting key 61f276929a2f15a0ad794c03282e032b
I20260929 14:42:56.767038 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0: Bootstrap starting.
I20260929 14:42:56.769490 10179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:56.769721 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.765451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.62:43973)
0929 14:42:56.766131 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.766146 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.766181 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:42:56.766849 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.766866 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.767270 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.767595 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.768385 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:56.768390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:56.768403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:56.768413 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:56.768471 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:56.768488 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:56.768612 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:56.768685 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 14:42:56.768961 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":135,"threads_started":1}
I20260929 14:42:56.770292 10179 log.cc:824] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:56.772277 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0: No bootstrap required, opened a new log
I20260929 14:42:56.776010 10179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260929 14:42:56.776250 10179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:56.776309 10179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5efcd93614284c14a30da2abdaa5d9f0, State: Initialized, Role: FOLLOWER
I20260929 14:42:56.776734 10179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260929 14:42:56.776918 10179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:56.776986 10179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:56.777083 10179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:56.778048 10179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260929 14:42:56.778398 10179 leader_election.cc:302] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5efcd93614284c14a30da2abdaa5d9f0; no voters:
I20260929 14:42:56.778718 10179 leader_election.cc:288] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:56.778807 10184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:56.778993 10184 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 5efcd93614284c14a30da2abdaa5d9f0, State: Running, Role: LEADER
I20260929 14:42:56.779301 10184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } }
I20260929 14:42:56.780067 10179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:56.781037 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5efcd93614284c14a30da2abdaa5d9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } }
I20260929 14:42:56.781078 10186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5efcd93614284c14a30da2abdaa5d9f0. Latest consensus state: current_term: 1 leader_uuid: "5efcd93614284c14a30da2abdaa5d9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5efcd93614284c14a30da2abdaa5d9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43973 } } }
I20260929 14:42:56.781152 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:56.781181 10186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:56.781503 10193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:56.784687 10193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:56.790602 10193 catalog_manager.cc:1347] Generated new cluster ID: b2b30a146d444d8b83fc32bcf01a10ee
I20260929 14:42:56.790689 10193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:56.811899 10193 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:56.812280 10193 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:56.816813 10193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5efcd93614284c14a30da2abdaa5d9f0: Generated new TSK 0
I20260929 14:42:56.817716 10193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:42:56.826109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:42:56.937239 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:56.937526 10203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:56.937567 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:56.941730 10203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:56.941790 10203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:56.941849 10203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:56.946697 10203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:56.947877 10203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:56.949385 10203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:56.959245 10203 server_base.cc:1034] running on GCE node
W20260929 14:42:56.959177 10208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.959172 10211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:56.959190 10209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:56.959908 10203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:56.960553 10203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:56.961721 10203 hybrid_clock.cc:648] HybridClock initialized: now 1790692976961700 us; error 31 us; skew 500 ppm
I20260929 14:42:56.963610 10203 webserver.cc:460] Webserver started at http://127.0.98.1:40279/ using document root <none> and password file <none>
I20260929 14:42:56.964169 10203 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:56.964233 10203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:56.964432 10203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:56.966706 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d44b49c4a824787815c1f84701771b7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "585c54ee01c1d02eb393af6a0d5e82eb"
server_key_iv: "d9d8c4516933ba3445a524773064fdab"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.967245 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d44b49c4a824787815c1f84701771b7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "585c54ee01c1d02eb393af6a0d5e82eb"
server_key_iv: "d9d8c4516933ba3445a524773064fdab"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.971143 10203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:56.973759 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.975126 10203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:56.975237 10203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "1d44b49c4a824787815c1f84701771b7"
format_stamp: "Formatted at 2026-09-29 14:42:56 on dist-test-slave-g8hp"
server_key: "585c54ee01c1d02eb393af6a0d5e82eb"
server_key_iv: "d9d8c4516933ba3445a524773064fdab"
server_key_version: "encryptionkey@0"
I20260929 14:42:56.975373 10203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:56.980079 10203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:56.980753 10203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:56.980952 10203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:56.981549 10203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:56.982466 10203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:56.982578 10203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.982686 10203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:56.982719 10203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:56.991528 10203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35857
I20260929 14:42:56.991571 10330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35857 every 8 connection(s)
I20260929 14:42:56.992475 10203 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:56.998607 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.994314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60549 (local address 127.0.98.62:43973)
0929 14:42:56.994512 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:56.994517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:56.995692 (+ 1175us) server_negotiation.cc:407] Connection header received
0929 14:42:56.996984 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:56.996987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:56.997026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:56.997105 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:56.998222 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:56.998226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:56.998228 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:56.998241 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:56.998261 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:56.998268 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:56.998371 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:56.998454 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 14:42:56.998494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 14:42:56.999508 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:56.994806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.1:60549)
0929 14:42:56.995527 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:56.995571 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:56.996720 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:56.997264 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:56.997271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:56.997362 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:56.997958 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:56.997960 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:56.998002 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:56.998073 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:56.998341 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:56.998347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:56.998351 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:42:56.998486 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 14:42:56.998699 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":125,"threads_started":1}
I20260929 14:42:57.000680 10331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260929 14:42:57.000931 10331 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:57.001333 10331 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260929 14:42:57.002123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10203
I20260929 14:42:57.002224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:57.002512 392 external_mini_cluster.cc:1428] Setting key 72767ec42bebfa0499b985402774a8c1
I20260929 14:42:57.002956 10143 ts_manager.cc:194] Registered new tserver with Master: 1d44b49c4a824787815c1f84701771b7 (127.0.98.1:35857)
I20260929 14:42:57.004467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:42:57.117460 10334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:57.117746 10334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:57.117787 10334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:57.121965 10334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:57.122035 10334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:57.122094 10334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:57.127059 10334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:57.128159 10334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:57.129628 10334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:57.139813 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:57.139813 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:57.139941 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:57.139993 10334 server_base.cc:1034] running on GCE node
I20260929 14:42:57.140645 10334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:57.141357 10334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:57.142550 10334 hybrid_clock.cc:648] HybridClock initialized: now 1790692977142529 us; error 34 us; skew 500 ppm
I20260929 14:42:57.144477 10334 webserver.cc:460] Webserver started at http://127.0.98.2:46205/ using document root <none> and password file <none>
I20260929 14:42:57.145139 10334 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:57.145212 10334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:57.145424 10334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:57.147559 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "a4f2107aa0494f94a24145f6bfacd7ba"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "0851270a823ce7967bfe0ec3e664bdb1"
server_key_iv: "07f1ee9593b9f447c6901609421cd413"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.148149 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a4f2107aa0494f94a24145f6bfacd7ba"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "0851270a823ce7967bfe0ec3e664bdb1"
server_key_iv: "07f1ee9593b9f447c6901609421cd413"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.151966 10334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:42:57.154507 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.155978 10334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:42:57.156101 10334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "a4f2107aa0494f94a24145f6bfacd7ba"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "0851270a823ce7967bfe0ec3e664bdb1"
server_key_iv: "07f1ee9593b9f447c6901609421cd413"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.156208 10334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:57.161051 10334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:57.161701 10334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:57.161867 10334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:57.162434 10334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:57.163355 10334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:57.163401 10334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.163463 10334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:57.163489 10334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.171670 10334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39251
I20260929 14:42:57.171713 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39251 every 8 connection(s)
I20260929 14:42:57.172577 10334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:57.179072 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:57.174510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43537 (local address 127.0.98.62:43973)
0929 14:42:57.174655 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:57.174659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:57.175901 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 14:42:57.177323 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:57.177326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:57.177366 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:57.177447 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:57.178579 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:57.178583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:57.178585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:57.178598 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:57.178618 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:57.178626 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:57.178705 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:57.178871 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:42:57.178909 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:42:57.180052 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:57.175016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.2:43537)
0929 14:42:57.175729 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:57.175775 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:57.177099 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:57.177601 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:57.177608 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:57.177704 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:57.178290 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:57.178292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:57.178336 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:57.178433 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:57.178739 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:57.178747 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:57.178752 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:42:57.178916 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 14:42:57.179129 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260929 14:42:57.181140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10334
I20260929 14:42:57.181263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:57.181363 10462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260929 14:42:57.181614 392 external_mini_cluster.cc:1428] Setting key 227b0d20a816cdbc51d424e9cc4e979b
I20260929 14:42:57.181651 10462 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:57.182109 10462 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260929 14:42:57.183080 10143 ts_manager.cc:194] Registered new tserver with Master: a4f2107aa0494f94a24145f6bfacd7ba (127.0.98.2:39251)
I20260929 14:42:57.183459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--builtin_ntp_servers=127.0.98.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 14:42:57.297199 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:57.297488 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:57.297534 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:57.301795 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:57.301870 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:57.301936 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:57.306893 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:57.307989 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:57.309443 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:57.319392 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:57.319392 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:57.319578 10465 server_base.cc:1034] running on GCE node
W20260929 14:42:57.319396 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:57.320083 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:57.320788 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:57.322022 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790692977321996 us; error 42 us; skew 500 ppm
I20260929 14:42:57.324100 10465 webserver.cc:460] Webserver started at http://127.0.98.3:44637/ using document root <none> and password file <none>
I20260929 14:42:57.324697 10465 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:57.324785 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:57.325042 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:57.327219 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "74dddfa62ba1439d94dade4fdf565dce"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "538d46083ee165dcb524349696f3508d"
server_key_iv: "e9df2fd824eacfab69c3f180215d6a20"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.327782 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "74dddfa62ba1439d94dade4fdf565dce"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "538d46083ee165dcb524349696f3508d"
server_key_iv: "e9df2fd824eacfab69c3f180215d6a20"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.331815 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:42:57.334749 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.336158 10465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:42:57.336275 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "74dddfa62ba1439d94dade4fdf565dce"
format_stamp: "Formatted at 2026-09-29 14:42:57 on dist-test-slave-g8hp"
server_key: "538d46083ee165dcb524349696f3508d"
server_key_iv: "e9df2fd824eacfab69c3f180215d6a20"
server_key_version: "encryptionkey@0"
I20260929 14:42:57.336392 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:57.341033 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:57.341609 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:57.341766 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:57.342280 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:57.343164 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:57.343209 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.343258 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:57.343286 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:57.352007 10465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40391
I20260929 14:42:57.352043 10592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40391 every 8 connection(s)
I20260929 14:42:57.352998 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:42:57.359565 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:57.354892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41713 (local address 127.0.98.62:43973)
0929 14:42:57.355128 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:57.355133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:57.356316 (+ 1183us) server_negotiation.cc:407] Connection header received
0929 14:42:57.357641 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:57.357644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:57.357682 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:57.357761 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:57.359043 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:57.359046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:57.359048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:42:57.359059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:57.359078 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:42:57.359087 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:57.359162 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:57.359325 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 14:42:57.359379 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 14:42:57.359613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10465
I20260929 14:42:57.359706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:57.359983 392 external_mini_cluster.cc:1428] Setting key 79a76c2214cb4ff69f0e1ebcbcd97aa7
I20260929 14:42:57.360627 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:57.355404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43973 (local address 127.0.98.3:41713)
0929 14:42:57.356153 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:57.356206 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:57.357425 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:57.358055 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:57.358062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:57.358162 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:57.358723 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:42:57.358725 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:57.358767 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:42:57.358840 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:57.359185 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:57.359193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:57.359198 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 14:42:57.359381 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 14:42:57.359617 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260929 14:42:57.361714 10593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43973
I20260929 14:42:57.362071 10593 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:57.362497 10593 heartbeater.cc:511] Master 127.0.98.62:43973 requested a full tablet report, sending...
I20260929 14:42:57.363596 10143 ts_manager.cc:194] Registered new tserver with Master: 74dddfa62ba1439d94dade4fdf565dce (127.0.98.3:40391)
I20260929 14:42:57.374156 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 14:42:57.377921 10182 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:42:57.377065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.62:43973)
0929 14:42:57.377215 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:57.377220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:57.377239 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:42:57.377287 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:57.377290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:57.377333 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:57.377425 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:57.377750 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56654: BlockingRecv error: recv got EOF from 127.0.0.1:56654 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:57.381381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10203
I20260929 14:42:57.387180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10334
I20260929 14:42:57.392330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10465
I20260929 14:42:57.396972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10112
2026-09-29T14:42:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (826 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:57 dist-test-slave-g8hp krb5kdc[10603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:57 dist-test-slave-g8hp krb5kdc[10603](info): set up 2 sockets
Sep 29 14:42:57 dist-test-slave-g8hp krb5kdc[10603](info): commencing operation
krb5kdc: starting...
W20260929 14:42:58.365626 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:42:58 dist-test-slave-g8hp krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:42:58.494248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:58.605248 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:58.605544 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:58.605595 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:58.609613 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:42:58.609676 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:58.609695 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:58.609710 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:42:58.609724 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:42:58.615109 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:58.616257 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:58.617312 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:58.626187 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:58.626210 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:58.626250 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:58.626376 10616 server_base.cc:1034] running on GCE node
I20260929 14:42:58.627267 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:58.628556 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:58.629772 10616 hybrid_clock.cc:648] HybridClock initialized: now 1790692978629758 us; error 57 us; skew 500 ppm
Sep 29 14:42:58 dist-test-slave-g8hp krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:58.632745 10616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:42:58.633719 10616 webserver.cc:460] Webserver started at http://127.0.98.62:38709/ using document root <none> and password file <none>
I20260929 14:42:58.634279 10616 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:58.634352 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:58.634526 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:58.636659 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "daa59c076d4a493e87f9d48f99dd99b9"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "64688186e90bd0795ddd2fe70e69c930"
server_key_iv: "a4ac7594a9bc1837e0478109d90e7f02"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.637239 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "daa59c076d4a493e87f9d48f99dd99b9"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "64688186e90bd0795ddd2fe70e69c930"
server_key_iv: "a4ac7594a9bc1837e0478109d90e7f02"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.641191 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:58.643764 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:58.645159 10616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:58.645282 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "daa59c076d4a493e87f9d48f99dd99b9"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "64688186e90bd0795ddd2fe70e69c930"
server_key_iv: "a4ac7594a9bc1837e0478109d90e7f02"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.645390 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:58.659879 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:58.663380 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:58.663561 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:58.670239 10616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44259
I20260929 14:42:58.670291 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44259 every 8 connection(s)
I20260929 14:42:58.671222 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:42:58.674125 10684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:58.680020 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: Bootstrap starting.
I20260929 14:42:58.680814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10616
I20260929 14:42:58.680956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:42:58.681192 392 external_mini_cluster.cc:1428] Setting key 4e42abacc321fa5377f705cd2443e31a
I20260929 14:42:58.682945 10684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:58.683666 10684 log.cc:824] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:58.685593 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: No bootstrap required, opened a new log
I20260929 14:42:58.689119 10684 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260929 14:42:58.689384 10684 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 29 14:42:58 dist-test-slave-g8hp krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:58.689451 10684 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa59c076d4a493e87f9d48f99dd99b9, State: Initialized, Role: FOLLOWER
I20260929 14:42:58.689988 10684 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260929 14:42:58.690148 10684 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:42:58.690217 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:42:58.690316 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:58.691368 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260929 14:42:58.691709 10684 leader_election.cc:302] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa59c076d4a493e87f9d48f99dd99b9; no voters:
I20260929 14:42:58.692023 10684 leader_election.cc:288] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:42:58.692143 10689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:58.692313 10689 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [term 1 LEADER]: Becoming Leader. State: Replica: daa59c076d4a493e87f9d48f99dd99b9, State: Running, Role: LEADER
I20260929 14:42:58.692617 10689 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260929 14:42:58.693405 10684 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:42:58.693856 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa59c076d4a493e87f9d48f99dd99b9. Latest consensus state: current_term: 1 leader_uuid: "daa59c076d4a493e87f9d48f99dd99b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } } }
I20260929 14:42:58.693974 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:58.694624 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa59c076d4a493e87f9d48f99dd99b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa59c076d4a493e87f9d48f99dd99b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } } }
I20260929 14:42:58.694717 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:42:58.695188 10693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:42:58.696192 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:58.682408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.0.98.62:44259)
0929 14:42:58.682852 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:58.682867 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:58.682904 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:42:58.683589 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:58.683606 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:58.684040 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:58.684373 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:58.685129 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.686166 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.686998 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.687473 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.687625 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:58.690106 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:58.690111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:58.690125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:58.690158 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:58.692079 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:58.692663 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:58.692668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:58.692675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:58.692838 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:58.693168 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:58.693171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:58.693173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:58.693694 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:58.694475 (+ 781us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:58.694655 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 14:42:58.694973 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":145,"threads_started":1}
I20260929 14:42:58.698647 10693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:42:58.704527 10693 catalog_manager.cc:1347] Generated new cluster ID: d07c94efc6554f9f8d66515056dfb74a
I20260929 14:42:58.704600 10693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:42:58.715070 10693 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:42:58.716042 10693 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:42:58.721678 10693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: Generated new TSK 0
I20260929 14:42:58.722563 10693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:42:58.760810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:58.876029 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:58.876382 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:58.876430 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:58.880805 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:58.880913 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:58.880975 10710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:42:58.886647 10710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:58.887737 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:58.888747 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:42:58.899389 10710 server_base.cc:1034] running on GCE node
W20260929 14:42:58.899322 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:58.899422 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:58.899322 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:58.900163 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:58.900876 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:58.902064 10710 hybrid_clock.cc:648] HybridClock initialized: now 1790692978902039 us; error 36 us; skew 500 ppm
Sep 29 14:42:58 dist-test-slave-g8hp krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:58.905141 10710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:42:58.906092 10710 webserver.cc:460] Webserver started at http://127.0.98.1:38007/ using document root <none> and password file <none>
I20260929 14:42:58.906711 10710 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:58.906785 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:58.907001 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:58.909152 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae94b68f26b5406fb84d0fc0315906f5"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "4640b6c1955ecf16b116b44f4a990f66"
server_key_iv: "7c99142b3bc5912d4beb8c4af68741bc"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.909719 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae94b68f26b5406fb84d0fc0315906f5"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "4640b6c1955ecf16b116b44f4a990f66"
server_key_iv: "7c99142b3bc5912d4beb8c4af68741bc"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.913887 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:42:58.916695 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:58.918468 10710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:58.918637 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "ae94b68f26b5406fb84d0fc0315906f5"
format_stamp: "Formatted at 2026-09-29 14:42:58 on dist-test-slave-g8hp"
server_key: "4640b6c1955ecf16b116b44f4a990f66"
server_key_iv: "7c99142b3bc5912d4beb8c4af68741bc"
server_key_version: "encryptionkey@0"
I20260929 14:42:58.918762 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:58.933401 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:58.937310 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:58.937515 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:58.938133 10710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:58.939045 10710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:58.939096 10710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:58.939150 10710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:58.939177 10710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:58.948014 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40189
I20260929 14:42:58.948040 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40189 every 8 connection(s)
I20260929 14:42:58.949019 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:42:58.957630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10710
I20260929 14:42:58.957731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:42:58.958011 392 external_mini_cluster.cc:1428] Setting key 6c6a9cebbf74e53c9b3c9e6560b3254c
Sep 29 14:42:58 dist-test-slave-g8hp krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:58.964046 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:58.951042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52755 (local address 127.0.98.62:44259)
0929 14:42:58.951176 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:58.951179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:58.952101 (+ 922us) server_negotiation.cc:407] Connection header received
0929 14:42:58.953470 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:58.953473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:58.953526 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:58.953605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:58.954896 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.955424 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.956666 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.957010 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.957097 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:58.960055 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:58.960060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:58.960062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:58.960090 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:58.961625 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:58.962324 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:58.962327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:58.962328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:58.962408 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:58.962786 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:58.962788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:58.962789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:58.962974 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:58.963043 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:58.963705 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 14:42:58.963832 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:42:58.965269 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:58.951370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44259 (local address 127.0.98.1:52755)
0929 14:42:58.951928 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:58.951975 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:58.953225 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:58.953779 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:58.953786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:58.954222 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:58.954750 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:58.954766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.955540 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.955543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:58.956545 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:58.956555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:58.957185 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:58.957188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:58.957382 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:58.958070 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:58.958092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:58.959853 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:58.961765 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:58.961773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:58.961795 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:58.962221 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:58.962504 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:58.962506 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:58.962508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:58.962699 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:58.963060 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:58.963067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:58.963438 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 14:42:58.963664 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 14:42:58.964059 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260929 14:42:58.966689 10839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44259
I20260929 14:42:58.966980 10839 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:58.967664 10839 heartbeater.cc:511] Master 127.0.98.62:44259 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:42:58.969630 10648 ts_manager.cc:194] Registered new tserver with Master: ae94b68f26b5406fb84d0fc0315906f5 (127.0.98.1:40189)
I20260929 14:42:58.970860 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:42:58.989455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:59.103381 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:59.103669 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:59.103713 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:59.108206 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:59.108305 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:59.108372 10844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:42:59.114214 10844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:59.115350 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:59.116416 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:59.127671 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:59.127671 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:59.127671 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:59.128104 10844 server_base.cc:1034] running on GCE node
I20260929 14:42:59.128669 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:59.129418 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:59.130599 10844 hybrid_clock.cc:648] HybridClock initialized: now 1790692979130569 us; error 42 us; skew 500 ppm
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:59.133701 10844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:42:59.134744 10844 webserver.cc:460] Webserver started at http://127.0.98.2:33727/ using document root <none> and password file <none>
I20260929 14:42:59.135335 10844 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:59.135408 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:59.135622 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:59.137923 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "f5c0e9f4cd944b7fadf24767583d18cb"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "98444fc6e5169d7433954c61734fc262"
server_key_iv: "e90bd611f03db620fb11cd798eaa5266"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.138597 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f5c0e9f4cd944b7fadf24767583d18cb"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "98444fc6e5169d7433954c61734fc262"
server_key_iv: "e90bd611f03db620fb11cd798eaa5266"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.142700 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 14:42:59.145485 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.146873 10844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:59.147068 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "f5c0e9f4cd944b7fadf24767583d18cb"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "98444fc6e5169d7433954c61734fc262"
server_key_iv: "e90bd611f03db620fb11cd798eaa5266"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.147177 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:59.173326 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:59.177014 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:59.177196 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:59.177831 10844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:59.178745 10844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:59.178794 10844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.178835 10844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:59.178849 10844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.188134 10844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41031
I20260929 14:42:59.188205 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41031 every 8 connection(s)
I20260929 14:42:59.189157 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:42:59.196635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10844
I20260929 14:42:59.196738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:42:59.197118 392 external_mini_cluster.cc:1428] Setting key b26e65eccf3cb75e19bf664b5965e848
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:59.203835 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.191115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57069 (local address 127.0.98.62:44259)
0929 14:42:59.191248 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.191252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.192215 (+ 963us) server_negotiation.cc:407] Connection header received
0929 14:42:59.193561 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.193564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.193621 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.193696 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:59.195097 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.195678 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.196955 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.197175 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.197266 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.200136 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:59.200139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:59.200141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:59.200167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:59.201568 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.202090 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.202093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.202095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.202176 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.202615 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.202617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.202618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.202804 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:59.202875 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.203599 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.203707 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 14:42:59.205097 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.191549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44259 (local address 127.0.98.2:57069)
0929 14:42:59.192064 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.192110 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.193344 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.193902 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.193909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.194334 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:59.194934 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.194955 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.195852 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.195856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.196814 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.196824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.197448 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.197455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.197738 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.198385 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:59.198408 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:59.199951 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:59.201687 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:59.201693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:59.201712 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:59.201992 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:59.202298 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:59.202302 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:59.202304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:59.202520 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:59.202904 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:59.202912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:59.203280 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.203528 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.203910 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260929 14:42:59.206451 10973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:42:59.206705 10973 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:59.207315 10973 heartbeater.cc:511] Master 127.0.98.62:44259 requested a full tablet report, sending...
I20260929 14:42:59.208616 10648 ts_manager.cc:194] Registered new tserver with Master: f5c0e9f4cd944b7fadf24767583d18cb (127.0.98.2:41031)
I20260929 14:42:59.209246 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:42:59.227839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:42:59.342379 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:42:59.342676 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:42:59.342720 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:42:59.346859 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:42:59.346935 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:42:59.346997 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:42:59.352480 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:42:59.353646 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:42:59.354643 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:42:59.364817 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:59.364930 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:42:59.364897 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:42:59.364921 10978 server_base.cc:1034] running on GCE node
I20260929 14:42:59.365622 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:42:59.366286 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:42:59.367460 10978 hybrid_clock.cc:648] HybridClock initialized: now 1790692979367414 us; error 65 us; skew 500 ppm
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:42:59.370288 10978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:42:59.371181 10978 webserver.cc:460] Webserver started at http://127.0.98.3:44231/ using document root <none> and password file <none>
I20260929 14:42:59.371723 10978 fs_manager.cc:359] Metadata directory not provided
I20260929 14:42:59.371783 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:42:59.371946 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:42:59.374071 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "099d1be44a604d8f91a2a65aeac74168"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "4fe534f4e2851de74bb9b3df2350c01a"
server_key_iv: "f308e76b69de9c9289cac3b4caa77ed0"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.374635 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "099d1be44a604d8f91a2a65aeac74168"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "4fe534f4e2851de74bb9b3df2350c01a"
server_key_iv: "f308e76b69de9c9289cac3b4caa77ed0"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.378623 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:42:59.381562 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.382886 10978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:42:59.382997 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "099d1be44a604d8f91a2a65aeac74168"
format_stamp: "Formatted at 2026-09-29 14:42:59 on dist-test-slave-g8hp"
server_key: "4fe534f4e2851de74bb9b3df2350c01a"
server_key_iv: "f308e76b69de9c9289cac3b4caa77ed0"
server_key_version: "encryptionkey@0"
I20260929 14:42:59.383096 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:42:59.409675 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:42:59.413022 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:42:59.413218 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:42:59.413866 10978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:42:59.414742 10978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:42:59.414799 10978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.414855 10978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:42:59.414880 10978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:42:59.423645 10978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33875
I20260929 14:42:59.423731 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33875 every 8 connection(s)
I20260929 14:42:59.424582 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:42:59.434798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 10978
I20260929 14:42:59.434914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:42:59.435192 392 external_mini_cluster.cc:1428] Setting key 65cf1edec8af37cd619399f5097aea30
I20260929 14:42:59.439216 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.426511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47035 (local address 127.0.98.62:44259)
0929 14:42:59.426698 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.426702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.427584 (+ 882us) server_negotiation.cc:407] Connection header received
0929 14:42:59.428973 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.428976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.429026 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.429102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:59.430680 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.431218 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.432490 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.432715 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.432798 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.435461 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:59.435465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:59.435467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:59.435496 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:59.437065 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.437696 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.437699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.437700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.437783 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.438087 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.438089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.438090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.438326 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:59.438402 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.438950 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.439042 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:42:59.440063 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.426949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44259 (local address 127.0.98.3:47035)
0929 14:42:59.427422 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.427469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.428700 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.429344 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.429351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.429850 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:42:59.430488 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.430504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.431382 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.431385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.432366 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.432376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.432856 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.432859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.433072 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.433749 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:42:59.433767 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:42:59.435268 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:42:59.437256 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:59.437262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:59.437281 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:59.437610 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:59.437861 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:42:59.437864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:42:59.437865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:42:59.438019 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:42:59.438439 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:42:59.438444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:42:59.438748 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.438919 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.439192 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260929 14:42:59.441232 11107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44259
I20260929 14:42:59.441445 11107 heartbeater.cc:464] Registering TS with master...
I20260929 14:42:59.441902 11107 heartbeater.cc:511] Master 127.0.98.62:44259 requested a full tablet report, sending...
I20260929 14:42:59.442894 10648 ts_manager.cc:194] Registered new tserver with Master: 099d1be44a604d8f91a2a65aeac74168 (127.0.98.3:33875)
I20260929 14:42:59.443462 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47035
I20260929 14:42:59.449771 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:42:59.459386 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.0.98.62:44259)
0929 14:42:59.451967 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.451972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.452158 (+ 186us) server_negotiation.cc:407] Connection header received
0929 14:42:59.452251 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.452254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.452299 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.452362 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:42:59.453115 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.453641 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.454654 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.454884 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.454975 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.456326 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:42:59.456330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:42:59.456332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:42:59.456358 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:42:59.457758 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.458165 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.458169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.458171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.458253 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:42:59.458499 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:42:59.458502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:42:59.458504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:42:59.458738 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:42:59.458856 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.459090 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.459216 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 14:42:59.462589 10648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:42:59.465783 10648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:42:59.480942 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.474917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.98.1:40189)
0929 14:42:59.475335 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.475342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.475549 (+ 207us) server_negotiation.cc:407] Connection header received
0929 14:42:59.475777 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.475784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.476011 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.476133 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:59.476570 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.477750 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.479408 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.480270 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.480378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.480530 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.480683 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.480781 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":117,"threads_started":1}
I20260929 14:42:59.480935 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.474736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40189 (local address 127.0.0.1:53356)
0929 14:42:59.475463 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.475476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.475617 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.476322 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.476326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.476345 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:59.476458 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.476462 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.477857 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.477860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.479277 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.479288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.480391 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.480394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.480485 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.480518 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.480617 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.480747 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"mutex_wait_us":68,"thread_start_us":80,"threads_started":1}
I20260929 14:42:59.481364 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.474643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41031 (local address 127.0.0.1:58206)
0929 14:42:59.475124 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.475189 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.475319 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.476817 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.476821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.476839 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:59.476942 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.476948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.478336 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.478340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.479302 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.479312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.480958 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.480962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.481057 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.481080 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.481157 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.481236 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":93,"threads_started":1}
I20260929 14:42:59.481508 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.475436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.0.98.2:41031)
0929 14:42:59.476183 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.476196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.476218 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:42:59.476295 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.476300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.476478 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.476636 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:59.477062 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.478072 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.479440 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.480270 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.480378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.480530 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.481275 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.481377 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":125,"threads_started":1}
I20260929 14:42:59.481988 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.475416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33875 (local address 127.0.0.1:50878)
0929 14:42:59.475845 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.475858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.475947 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.476614 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.476617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.476635 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:59.476719 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.476725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.477730 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.477733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.479277 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.479288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.481631 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.481635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.481728 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.481751 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.481806 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.481873 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":72,"thread_start_us":89,"threads_started":1}
I20260929 14:42:59.482496 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.475818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.0.98.3:33875)
0929 14:42:59.476230 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.476242 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.476260 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:42:59.476318 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.476322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.476479 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.476627 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:59.476793 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.477609 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.480596 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.481306 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.481401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.482223 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.482274 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.482364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":132,"threads_started":1}
I20260929 14:42:59.484268 10907 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 (DEFAULT_TABLE table=test-table [id=ffaa94705d3d49d68b9cebc90b5299dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:59.484392 10773 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 (DEFAULT_TABLE table=test-table [id=ffaa94705d3d49d68b9cebc90b5299dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:59.485229 10907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf189bb95b4d09b0e689c525a446c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:59.485057 11041 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 (DEFAULT_TABLE table=test-table [id=ffaa94705d3d49d68b9cebc90b5299dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:42:59.485426 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf189bb95b4d09b0e689c525a446c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:59.485960 11041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf189bb95b4d09b0e689c525a446c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:42:59.491672 11124 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Bootstrap starting.
I20260929 14:42:59.493161 11126 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Bootstrap starting.
I20260929 14:42:59.494300 11124 tablet_bootstrap.cc:654] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:59.494841 11125 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Bootstrap starting.
I20260929 14:42:59.495090 11124 log.cc:824] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:59.496064 11126 tablet_bootstrap.cc:654] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:59.496789 11125 tablet_bootstrap.cc:654] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:42:59.497400 11126 log.cc:824] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:59.497529 11125 log.cc:824] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:42:59.497807 11124 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: No bootstrap required, opened a new log
I20260929 14:42:59.498009 11124 ts_tablet_manager.cc:1397] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260929 14:42:59.499236 11125 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: No bootstrap required, opened a new log
I20260929 14:42:59.499416 11125 ts_tablet_manager.cc:1397] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:42:59.499430 11126 tablet_bootstrap.cc:492] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: No bootstrap required, opened a new log
I20260929 14:42:59.499598 11126 ts_tablet_manager.cc:1397] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:42:59.502079 11124 raft_consensus.cc:348] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.502393 11124 raft_consensus.cc:374] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:59.502478 11124 raft_consensus.cc:729] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5c0e9f4cd944b7fadf24767583d18cb, State: Initialized, Role: FOLLOWER
I20260929 14:42:59.502873 11125 raft_consensus.cc:348] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.503010 11124 consensus_queue.cc:260] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.503144 11125 raft_consensus.cc:374] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:59.503226 11125 raft_consensus.cc:729] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae94b68f26b5406fb84d0fc0315906f5, State: Initialized, Role: FOLLOWER
I20260929 14:42:59.503424 11126 raft_consensus.cc:348] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.503698 11126 raft_consensus.cc:374] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:42:59.503662 11125 consensus_queue.cc:260] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.503764 11126 raft_consensus.cc:729] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099d1be44a604d8f91a2a65aeac74168, State: Initialized, Role: FOLLOWER
I20260929 14:42:59.504168 11124 ts_tablet_manager.cc:1428] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 14:42:59.504216 11126 consensus_queue.cc:260] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.504567 10973 heartbeater.cc:503] Master 127.0.98.62:44259 was elected leader, sending a full tablet report...
I20260929 14:42:59.504647 11125 ts_tablet_manager.cc:1428] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260929 14:42:59.504943 11107 heartbeater.cc:503] Master 127.0.98.62:44259 was elected leader, sending a full tablet report...
I20260929 14:42:59.504943 10839 heartbeater.cc:503] Master 127.0.98.62:44259 was elected leader, sending a full tablet report...
I20260929 14:42:59.505244 11126 ts_tablet_manager.cc:1428] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260929 14:42:59.675570 11108 tablet.cc:2369] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:59.690325 10974 tablet.cc:2369] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:42:59.700378 10840 tablet.cc:2369] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:42:59.714980 11132 raft_consensus.cc:482] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:42:59.715255 11132 raft_consensus.cc:504] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.716472 11132 leader_election.cc:288] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5c0e9f4cd944b7fadf24767583d18cb (127.0.98.2:41031), ae94b68f26b5406fb84d0fc0315906f5 (127.0.98.1:40189)
I20260929 14:42:59.721652 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.717004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36401 (local address 127.0.98.1:40189)
0929 14:42:59.717239 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.717244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.717397 (+ 153us) server_negotiation.cc:407] Connection header received
0929 14:42:59.717501 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.717503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.717551 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.717611 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:59.717846 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.718561 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.720360 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.721038 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.721263 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.721317 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.721420 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.721491 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 14:42:59.721683 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.716834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54097 (local address 127.0.98.2:41031)
0929 14:42:59.716984 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.716987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.717277 (+ 290us) server_negotiation.cc:407] Connection header received
0929 14:42:59.717330 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.717333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.717378 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.717451 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:42:59.717741 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.718509 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.720447 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.721110 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.721209 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.721282 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.721497 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.721571 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 14:42:59.721652 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.716736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41031 (local address 127.0.98.3:54097)
0929 14:42:59.717075 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.717091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.717193 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.717483 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.717486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.717501 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:59.717621 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.717628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.718675 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.718679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.720192 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.720304 (+ 112us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.721284 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.721289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.721391 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.721409 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.721469 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.721524 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":70}
I20260929 14:42:59.721658 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.716849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40189 (local address 127.0.98.3:36401)
0929 14:42:59.717308 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.717320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:42:59.717417 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:42:59.717642 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:42:59.717645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:42:59.717660 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:42:59.717758 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.717763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.718675 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.718679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.720192 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:42:59.720206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.721173 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.721178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:42:59.721286 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.721316 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 14:42:59.721405 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:42:59.721486 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":139,"thread_start_us":121,"threads_started":1}
I20260929 14:42:59.722354 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baf189bb95b4d09b0e689c525a446c6" candidate_uuid: "099d1be44a604d8f91a2a65aeac74168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae94b68f26b5406fb84d0fc0315906f5" is_pre_election: true
I20260929 14:42:59.722380 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baf189bb95b4d09b0e689c525a446c6" candidate_uuid: "099d1be44a604d8f91a2a65aeac74168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" is_pre_election: true
I20260929 14:42:59.722681 10793 raft_consensus.cc:2454] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099d1be44a604d8f91a2a65aeac74168 in term 0.
I20260929 14:42:59.722680 10927 raft_consensus.cc:2454] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099d1be44a604d8f91a2a65aeac74168 in term 0.
I20260929 14:42:59.723162 10996 leader_election.cc:302] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099d1be44a604d8f91a2a65aeac74168, f5c0e9f4cd944b7fadf24767583d18cb; no voters:
I20260929 14:42:59.723461 11132 raft_consensus.cc:2781] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:42:59.723549 11132 raft_consensus.cc:482] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:42:59.723583 11132 raft_consensus.cc:3037] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:59.724669 11132 raft_consensus.cc:504] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.725054 11132 leader_election.cc:288] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [CANDIDATE]: Term 1 election: Requested vote from peers f5c0e9f4cd944b7fadf24767583d18cb (127.0.98.2:41031), ae94b68f26b5406fb84d0fc0315906f5 (127.0.98.1:40189)
I20260929 14:42:59.725469 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baf189bb95b4d09b0e689c525a446c6" candidate_uuid: "099d1be44a604d8f91a2a65aeac74168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae94b68f26b5406fb84d0fc0315906f5"
I20260929 14:42:59.725561 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1baf189bb95b4d09b0e689c525a446c6" candidate_uuid: "099d1be44a604d8f91a2a65aeac74168" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5c0e9f4cd944b7fadf24767583d18cb"
I20260929 14:42:59.725612 10793 raft_consensus.cc:3037] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:59.725744 10927 raft_consensus.cc:3037] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:42:59.726966 10793 raft_consensus.cc:2454] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099d1be44a604d8f91a2a65aeac74168 in term 1.
I20260929 14:42:59.726966 10927 raft_consensus.cc:2454] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099d1be44a604d8f91a2a65aeac74168 in term 1.
I20260929 14:42:59.727299 10997 leader_election.cc:302] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099d1be44a604d8f91a2a65aeac74168, ae94b68f26b5406fb84d0fc0315906f5; no voters:
I20260929 14:42:59.727469 11132 raft_consensus.cc:2781] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:42:59.727696 11132 raft_consensus.cc:686] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 1 LEADER]: Becoming Leader. State: Replica: 099d1be44a604d8f91a2a65aeac74168, State: Running, Role: LEADER
I20260929 14:42:59.728063 11132 consensus_queue.cc:237] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } }
I20260929 14:42:59.731487 10642 catalog_manager.cc:5495] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 reported cstate change: term changed from 0 to 1, leader changed from <none> to 099d1be44a604d8f91a2a65aeac74168 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "099d1be44a604d8f91a2a65aeac74168" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "099d1be44a604d8f91a2a65aeac74168" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33875 } health_report { overall_health: HEALTHY } } }
I20260929 14:42:59.774931 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.771294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.0.98.3:33875)
0929 14:42:59.771427 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:42:59.771431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:42:59.771445 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:42:59.771576 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:42:59.771579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:42:59.771636 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:42:59.771736 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:42:59.771984 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.772536 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.773485 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:42:59.773701 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:42:59.773788 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:42:59.774097 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:42:59.774531 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:42:59.774642 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:42:59.774716 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 14:42:59.774775 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:42:59.781121 10927 raft_consensus.cc:1261] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 1 FOLLOWER]: Refusing update from remote peer 099d1be44a604d8f91a2a65aeac74168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:59.781124 10793 raft_consensus.cc:1261] T 1baf189bb95b4d09b0e689c525a446c6 P ae94b68f26b5406fb84d0fc0315906f5 [term 1 FOLLOWER]: Refusing update from remote peer 099d1be44a604d8f91a2a65aeac74168: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:42:59.781808 11134 consensus_queue.cc:1035] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae94b68f26b5406fb84d0fc0315906f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:59.781996 11132 consensus_queue.cc:1035] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5c0e9f4cd944b7fadf24767583d18cb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:42:59.790849 11138 mvcc.cc:205] Tried to move back new op lower bound from 7334678445174345728 to 7334678444968222720. Current Snapshot: MvccSnapshot[applied={T|T < 7334678445174345728}]
I20260929 14:42:59.791019 11139 mvcc.cc:205] Tried to move back new op lower bound from 7334678445174345728 to 7334678444968222720. Current Snapshot: MvccSnapshot[applied={T|T < 7334678445174345728}]
I20260929 14:42:59.803726 10648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:59.803916 10648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:42:59.806316 10648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daa59c076d4a493e87f9d48f99dd99b9: Sending DeleteTablet for 3 replicas of tablet 1baf189bb95b4d09b0e689c525a446c6
I20260929 14:42:59.807164 11041 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50878
I20260929 14:42:59.807164 10773 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53356
I20260929 14:42:59.807720 10907 tablet_service.cc:1514] Processing DeleteTablet for tablet 1baf189bb95b4d09b0e689c525a446c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:42:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58206
I20260929 14:42:59.807853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10710
I20260929 14:42:59.808506 11147 tablet_replica.cc:330] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: stopping tablet replica
I20260929 14:42:59.808506 11149 tablet_replica.cc:330] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: stopping tablet replica
I20260929 14:42:59.808892 11147 raft_consensus.cc:2229] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:42:59.808889 11149 raft_consensus.cc:2229] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:42:59.809325 11149 raft_consensus.cc:2258] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:59.809304 11147 raft_consensus.cc:2258] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:42:59.810906 11147 ts_tablet_manager.cc:1905] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:59.810906 11149 ts_tablet_manager.cc:1905] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:42:59.813721 11147 ts_tablet_manager.cc:1918] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:59.813721 11149 ts_tablet_manager.cc:1918] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:42:59.813833 11147 log.cc:1196] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/1baf189bb95b4d09b0e689c525a446c6
I20260929 14:42:59.814020 11149 log.cc:1196] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/1baf189bb95b4d09b0e689c525a446c6
I20260929 14:42:59.814157 11147 ts_tablet_manager.cc:1939] T 1baf189bb95b4d09b0e689c525a446c6 P 099d1be44a604d8f91a2a65aeac74168: Deleting consensus metadata
I20260929 14:42:59.814463 11149 ts_tablet_manager.cc:1939] T 1baf189bb95b4d09b0e689c525a446c6 P f5c0e9f4cd944b7fadf24767583d18cb: Deleting consensus metadata
I20260929 14:42:59.815598 10634 catalog_manager.cc:4841] TS f5c0e9f4cd944b7fadf24767583d18cb (127.0.98.2:41031): tablet 1baf189bb95b4d09b0e689c525a446c6 (table test-table [id=ffaa94705d3d49d68b9cebc90b5299dc]) successfully deleted
I20260929 14:42:59.815800 10634 catalog_manager.cc:4841] TS 099d1be44a604d8f91a2a65aeac74168 (127.0.98.3:33875): tablet 1baf189bb95b4d09b0e689c525a446c6 (table test-table [id=ffaa94705d3d49d68b9cebc90b5299dc]) successfully deleted
W20260929 14:42:59.818374 10635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40189 (error 108)
I20260929 14:42:59.818427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10844
I20260929 14:42:59.822063 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:42:59.821740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40189 (local address 127.0.0.1:53372)
0929 14:42:59.821885 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 14:42:59.821964 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 14:42:59.822276 10635 catalog_manager.cc:4584] TS ae94b68f26b5406fb84d0fc0315906f5 (127.0.98.1:40189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1baf189bb95b4d09b0e689c525a446c6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40189: connect: Connection refused (error 111)
I20260929 14:42:59.823989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10978
I20260929 14:42:59.829421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 10616
2026-09-29T14:42:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2433 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[11153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[11153](info): set up 2 sockets
Sep 29 14:42:59 dist-test-slave-g8hp krb5kdc[11153](info): commencing operation
krb5kdc: starting...
W20260929 14:43:00.797719 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:43:00 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790692980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:43:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:43:00.972163 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:43:03.845 UTC [11183] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:43:03.846 UTC [11183] LOG: listening on IPv4 address "127.0.98.20", port 35543
2026-09-29 14:43:03.850 UTC [11183] LOG: listening on Unix socket "/tmp/.s.PGSQL.35543"
2026-09-29 14:43:03.879 UTC [11186] LOG: database system was shut down at 2026-09-29 14:43:02 UTC
2026-09-29 14:43:03.883 UTC [11183] LOG: database system is ready to accept connections
I20260929 14:43:04.761613 392 mini_postgres.cc:96] Postgres bound to 35543
2026-09-29 14:43:04.767 UTC [11194] FATAL: database "slave" does not exist
127.0.98.20:35543 - accepting connections
I20260929 14:43:04.768617 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:43:04.784222 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:43:05.380414 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:43:05.380515 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin
2026-09-29 14:43:05,653 [I] DB FLAVOR :POSTGRES
2026-09-29 14:43:05,654 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:43:05,654 [I] Checking connection..
2026-09-29 14:43:05,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:43:05,913 [I] Checking connection passed.
2026-09-29 14:43:05,914 [I] --------- Verifying version history table ---------
2026-09-29 14:43:05,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:06,174 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:43:06,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:06,435 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:43:06,435 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:43:06,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:43:06.693 UTC [11258] WARNING: there is no transaction in progress
2026-09-29 14:43:06,699 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:43:06,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:06,945 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:43:06,946 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:43:06,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:43:07,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:43:07,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g8hp.c.gcp-upstream.internal','N') ;"
2026-09-29 14:43:07,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:43:07,940 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:43:07,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:43:08,209 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:43:08,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:43:08,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:43:08,701 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:43:08,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:43:09.778 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.796 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.815 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.834 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.852 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.872 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.937 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.946 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.962 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.975 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:09.988 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10.006 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10.017 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10.026 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10.037 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10.046 UTC [11366] WARNING: there is no transaction in progress
2026-09-29 14:43:10,871 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:43:10,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:43:11,107 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:43:11,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:43:11,348 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:43:11,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:43:11,597 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:43:11,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:43:11,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:43:12,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g8hp.c.gcp-upstream.internal';"
2026-09-29 14:43:12,373 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:43:12,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:43:12,619 [I] DB_PATCHES have already been applied
I20260929 14:43:12.625448 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknwEhRJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:43:12.625517 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:43:12.628825 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33473
Sep 29, 2026 2:43:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:43:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:43:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:43:13.644568 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.876s user 0.000s sys 0.010s
Sep 29, 2026 2:43:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:43:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 2:43:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:43:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:43:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:43:34.971837 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:43:35 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693015, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:43:35.074309 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:43:35.083856 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:43:36.159646 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:43:36.159732 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms
I20260929 14:43:36.159825 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35543
2026-09-29 14:43:36,338 [I] DB FLAVOR :POSTGRES
2026-09-29 14:43:36,338 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:43:36,338 [I] Checking connection
2026-09-29 14:43:36,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:43:36,604 [I] connection success
2026-09-29 14:43:36,605 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:43:36,605 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:43:36,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:43:36,867 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:43:36,867 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:43:36,868 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:43:36,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35543/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:43:37,130 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:43:37.623307 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:43:37.793839 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:43:37.834458 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:43:37 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693015, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:43:38.207965 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:43:38.208133 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:43:38.208158 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:43:38.211596 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11546
I20260929 14:43:38.211736 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45343
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:43:40.300523 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.226s user 0.006s sys 0.155s
I20260929 14:43:40.300649 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:43:40.300765 392 mini_ranger_kms.cc:401] 127.0.98.20:45343/kms/v1/keys
Sep 29 14:43:40 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:43:40 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:43:41 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693020, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:43:43 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:43:43.648313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36735
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45343/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:43:43.761850 11597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:43:43.762142 11597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:43:43.762190 11597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:43:43.766456 11597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:43:43.766635 11597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:43:43.766685 11597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:43:43.766718 11597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:43:43.766749 11597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:43:43.772517 11597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45343/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36735
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:43:43.773692 11597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:43:43.774801 11597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:43:43.783849 11602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:43:43.783936 11605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:43.783911 11597 server_base.cc:1034] running on GCE node
W20260929 14:43:43.783861 11603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:43.784629 11597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:43:43.785831 11597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:43:43.786988 11597 hybrid_clock.cc:648] HybridClock initialized: now 1790693023786968 us; error 31 us; skew 500 ppm
Sep 29 14:43:43 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:43:43.795769 11597 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:43:43.796728 11597 webserver.cc:460] Webserver started at http://127.0.98.62:34331/ using document root <none> and password file <none>
I20260929 14:43:43.797318 11597 fs_manager.cc:359] Metadata directory not provided
I20260929 14:43:43.797380 11597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:43:43.797555 11597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:43:43 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:43:43.856172 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "136ba8a7867c493eb73606e88865e26c"
format_stamp: "Formatted at 2026-09-29 14:43:43 on dist-test-slave-g8hp"
server_key: "5979a9ab869f1d25a0ae28eadd5cd668"
server_key_iv: "18ca5ba34955128a1144317fc3f90b50"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:43.856845 11597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "136ba8a7867c493eb73606e88865e26c"
format_stamp: "Formatted at 2026-09-29 14:43:43 on dist-test-slave-g8hp"
server_key: "5979a9ab869f1d25a0ae28eadd5cd668"
server_key_iv: "18ca5ba34955128a1144317fc3f90b50"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:43.862077 11597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260929 14:43:43.880167 11612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:43.881708 11597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:43:43.881842 11597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "136ba8a7867c493eb73606e88865e26c"
format_stamp: "Formatted at 2026-09-29 14:43:43 on dist-test-slave-g8hp"
server_key: "5979a9ab869f1d25a0ae28eadd5cd668"
server_key_iv: "18ca5ba34955128a1144317fc3f90b50"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:43.881970 11597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:43:43.900853 11597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:43:43.903877 11597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:43:43.904479 11597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:43:43.911983 11597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36735
I20260929 14:43:43.912030 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36735 every 8 connection(s)
I20260929 14:43:43.913106 11597 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:43:43.915915 11665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:43:43.917003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 11597
I20260929 14:43:43.917132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:43:43.917418 392 external_mini_cluster.cc:1428] Setting key 73538381acb5370f8a8402c0f776fc42
I20260929 14:43:43.923615 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: Bootstrap starting.
I20260929 14:43:43.926313 11665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: Neither blocks nor log segments found. Creating new log.
I20260929 14:43:43.927146 11665 log.cc:824] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:43:43 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:43:43.929534 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: No bootstrap required, opened a new log
I20260929 14:43:43.932966 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:43.918998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.98.62:36735)
0929 14:43:43.919462 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:43.919480 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:43.919526 (+ 46us) server_negotiation.cc:407] Connection header received
0929 14:43:43.920438 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:43.920480 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:43.920998 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:43.921420 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:43:43.922269 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:43.923341 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:43.924458 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:43.925022 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:43.925153 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:43.927879 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:43:43.927885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:43:43.927902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:43:43.927932 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:43:43.930112 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:43.930591 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:43.930596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:43.930601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:43.930704 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:43.930951 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:43.930953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:43.930954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:43.931388 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:43:43.931538 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:43.931839 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 14:43:43.932087 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260929 14:43:43.932992 11665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260929 14:43:43.933322 11665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:43:43.933395 11665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136ba8a7867c493eb73606e88865e26c, State: Initialized, Role: FOLLOWER
I20260929 14:43:43.933879 11665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260929 14:43:43.934039 11665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:43:43.934105 11665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:43:43.934232 11665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:43:43.935168 11665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260929 14:43:43.935539 11665 leader_election.cc:302] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 136ba8a7867c493eb73606e88865e26c; no voters:
I20260929 14:43:43.935843 11665 leader_election.cc:288] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:43:43.935982 11670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:43:43.936193 11670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [term 1 LEADER]: Becoming Leader. State: Replica: 136ba8a7867c493eb73606e88865e26c, State: Running, Role: LEADER
I20260929 14:43:43.936559 11670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } }
I20260929 14:43:43.937054 11665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:43:43.938225 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 136ba8a7867c493eb73606e88865e26c. Latest consensus state: current_term: 1 leader_uuid: "136ba8a7867c493eb73606e88865e26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } } }
I20260929 14:43:43.938351 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [sys.catalog]: This master's current role is: LEADER
I20260929 14:43:43.938688 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "136ba8a7867c493eb73606e88865e26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "136ba8a7867c493eb73606e88865e26c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36735 } } }
I20260929 14:43:43.938773 11680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:43:43.938866 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c [sys.catalog]: This master's current role is: LEADER
I20260929 14:43:43.939157 11665 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:43:43.944793 11680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:43:43.951596 11680 catalog_manager.cc:1347] Generated new cluster ID: dfdc8846117048358b1ea40c1965f550
I20260929 14:43:43.951663 11680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:43:43.962306 11680 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:43:43.964154 11680 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:43:43.986847 11680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: Generated new TSK 0
I20260929 14:43:43.987666 11680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:43:44.692420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45343/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:43:44.823612 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:43:44.823895 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:43:44.823938 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:43:44.828186 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:43:44.828266 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:43:44.828356 11710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:43:44.835319 11710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45343/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:43:44.837024 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:43:44.838595 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:43:44.850361 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:43:44.850797 11720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:44.850886 11710 server_base.cc:1034] running on GCE node
W20260929 14:43:44.853513 11718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:44.854091 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:43:44.854790 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:43:44.855973 11710 hybrid_clock.cc:648] HybridClock initialized: now 1790693024855944 us; error 43 us; skew 500 ppm
Sep 29 14:43:44 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:43:44.858732 11710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:43:44.859714 11710 webserver.cc:460] Webserver started at http://127.0.98.1:42183/ using document root <none> and password file <none>
I20260929 14:43:44.860273 11710 fs_manager.cc:359] Metadata directory not provided
I20260929 14:43:44.860339 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:43:44.860538 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:43:44 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:43:44.882637 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8ecc59c176b413294325123d5f8f06f"
format_stamp: "Formatted at 2026-09-29 14:43:44 on dist-test-slave-g8hp"
server_key: "191eb549c27ea862076d2bf17033e13d"
server_key_iv: "da5d0be936eeef52e9680f133cfde80a"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:44.885432 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8ecc59c176b413294325123d5f8f06f"
format_stamp: "Formatted at 2026-09-29 14:43:44 on dist-test-slave-g8hp"
server_key: "191eb549c27ea862076d2bf17033e13d"
server_key_iv: "da5d0be936eeef52e9680f133cfde80a"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:44.891490 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260929 14:43:44.925292 11727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:44.927168 11710 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.000s sys 0.004s
I20260929 14:43:44.927399 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "f8ecc59c176b413294325123d5f8f06f"
format_stamp: "Formatted at 2026-09-29 14:43:44 on dist-test-slave-g8hp"
server_key: "191eb549c27ea862076d2bf17033e13d"
server_key_iv: "da5d0be936eeef52e9680f133cfde80a"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:44.927640 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:43:44.947913 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:43:44.951989 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:43:44.952167 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:43:44.952780 11710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:43:44.953675 11710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:43:44.953722 11710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:44.953768 11710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:43:44.953796 11710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:44.963913 11710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41735
I20260929 14:43:44.965226 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:43:44.971278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 11710
I20260929 14:43:44.971421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:43:44.971719 392 external_mini_cluster.cc:1428] Setting key 33349f63e85482482d4701db5a19cb17
I20260929 14:43:44.973122 11842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41735 every 8 connection(s)
Sep 29 14:43:44 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:43:44.985106 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:44.970016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56971 (local address 127.0.98.62:36735)
0929 14:43:44.971652 (+ 1636us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:44.971661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:44.971681 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:43:44.972435 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:44.972438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:44.972492 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:44.972584 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:43:44.974206 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:44.974968 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:44.976708 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:44.977161 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:44.977244 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:44.980450 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:43:44.980454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:43:44.980457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:43:44.980487 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:43:44.982383 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:44.983056 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:44.983059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:44.983061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:44.983159 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:44.983602 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:44.983606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:44.983607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:44.983865 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:43:44.983940 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:44.984727 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 14:43:44.984903 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1549,"thread_start_us":113,"threads_started":1}
I20260929 14:43:44.987457 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:44.968735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.1:56971)
0929 14:43:44.970512 (+ 1777us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:44.970588 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:44.972088 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:44.972924 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:44.972933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:44.973315 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:43:44.974037 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:44.974058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:44.975123 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:44.975127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:44.976565 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:44.976577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:44.977469 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:44.977473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:44.977694 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:44.978402 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:43:44.978422 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:43:44.980235 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:43:44.982549 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:44.982555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:44.982584 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:44.982934 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:44.983270 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:44.983274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:44.983276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:44.983464 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:44.984026 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:43:44.984033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:43:44.984412 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 14:43:44.985840 (+ 1428us) client_negotiation.cc:240] Negotiation successful
0929 14:43:44.986257 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"thread_start_us":165,"threads_started":1}
I20260929 14:43:44.989174 11843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260929 14:43:44.989418 11843 heartbeater.cc:464] Registering TS with master...
I20260929 14:43:44.989877 11843 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260929 14:43:44.991334 11629 ts_manager.cc:194] Registered new tserver with Master: f8ecc59c176b413294325123d5f8f06f (127.0.98.1:41735)
I20260929 14:43:44.992455 11629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56971
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:43:45.029341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45343/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:43:45.206758 11849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:43:45.207242 11849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:43:45.207355 11849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:43:45.212826 11849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:43:45.213017 11849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:43:45.213160 11849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:43:45.221520 11849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45343/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:43:45.223174 11849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:43:45.224223 11849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:43:45.235780 11858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:43:45.235952 11856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:43:45.235792 11855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:45.236485 11849 server_base.cc:1034] running on GCE node
I20260929 14:43:45.237046 11849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:43:45.237917 11849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:43:45.239166 11849 hybrid_clock.cc:648] HybridClock initialized: now 1790693025239135 us; error 48 us; skew 500 ppm
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:43:45.242363 11849 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:43:45.243357 11849 webserver.cc:460] Webserver started at http://127.0.98.2:36397/ using document root <none> and password file <none>
I20260929 14:43:45.244071 11849 fs_manager.cc:359] Metadata directory not provided
I20260929 14:43:45.244144 11849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:43:45.244361 11849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:43:45.261104 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "3170618b4fb24a798d45caf22db4abbe"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "8efd237a905c954c37b91ec387670855"
server_key_iv: "c8e4beb422799ae7d1ed91bd4e1ef270"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.262064 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3170618b4fb24a798d45caf22db4abbe"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "8efd237a905c954c37b91ec387670855"
server_key_iv: "c8e4beb422799ae7d1ed91bd4e1ef270"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.267572 11849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:43:45.289882 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:45.291739 11849 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260929 14:43:45.291886 11849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "3170618b4fb24a798d45caf22db4abbe"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "8efd237a905c954c37b91ec387670855"
server_key_iv: "c8e4beb422799ae7d1ed91bd4e1ef270"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.292026 11849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:43:45.315950 11849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:43:45.319698 11849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:43:45.319988 11849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:43:45.320777 11849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:43:45.322319 11849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:43:45.322460 11849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 14:43:45.322614 11849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:43:45.322692 11849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:45.334748 11849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40135
I20260929 14:43:45.336088 11849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:43:45.345077 11978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40135 every 8 connection(s)
I20260929 14:43:45.345607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 11849
I20260929 14:43:45.345842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:43:45.346338 392 external_mini_cluster.cc:1428] Setting key a4d70950ba76bf661d9334e9ad4d227f
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:43:45.364718 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.347114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47001 (local address 127.0.98.62:36735)
0929 14:43:45.347255 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.347259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.350465 (+ 3206us) server_negotiation.cc:407] Connection header received
0929 14:43:45.350520 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.350523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.350576 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.350655 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:43:45.352176 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.353005 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.354746 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.355089 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.355187 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.359442 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:43:45.359446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:43:45.359450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:43:45.359484 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:43:45.361325 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.362745 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.362749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.362751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.362846 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.363319 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.363322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.363324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.363576 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:43:45.363683 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.364376 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.364525 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:43:45.365974 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.347881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.2:47001)
0929 14:43:45.348519 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.348590 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.349639 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.350878 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.350889 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.351267 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:43:45.352024 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.352048 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.353145 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.353148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.354615 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.354627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.356579 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.356584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.356809 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.357561 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:43:45.357584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:43:45.359244 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:43:45.361446 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:45.361453 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:45.361475 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:45.361847 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:45.362941 (+ 1094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:45.362945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:45.362947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:45.363205 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:45.363683 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:43:45.363691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:43:45.364069 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.364299 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.364661 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":173,"threads_started":1}
I20260929 14:43:45.367213 11979 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260929 14:43:45.367475 11979 heartbeater.cc:464] Registering TS with master...
I20260929 14:43:45.368098 11979 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260929 14:43:45.369299 11629 ts_manager.cc:194] Registered new tserver with Master: 3170618b4fb24a798d45caf22db4abbe (127.0.98.2:40135)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:43:45.369985 11629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:43:45.396687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45343/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:43:45.561369 11984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:43:45.561764 11984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:43:45.561828 11984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:43:45.568059 11984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:43:45.568264 11984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:43:45.568387 11984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:43:45.576686 11984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45343/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:43:45.578204 11984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:43:45.579595 11984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:43:45.592319 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:45.592483 11984 server_base.cc:1034] running on GCE node
W20260929 14:43:45.592535 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:43:45.593039 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:43:45.593593 11984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:43:45.594406 11984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:43:45.609908 11984 hybrid_clock.cc:648] HybridClock initialized: now 1790693025609794 us; error 108 us; skew 500 ppm
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:43:45.614048 11984 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:43:45.615442 11984 webserver.cc:460] Webserver started at http://127.0.98.3:34639/ using document root <none> and password file <none>
I20260929 14:43:45.616191 11984 fs_manager.cc:359] Metadata directory not provided
I20260929 14:43:45.616278 11984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:43:45.616497 11984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:43:45.652334 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "dc6defe6589e491fab3950f99242ce1b"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "4adb0f05024ed98f9827a9321aa7533c"
server_key_iv: "0f79bc2767507bc84e99a6ec7d1be1cb"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.659215 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dc6defe6589e491fab3950f99242ce1b"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "4adb0f05024ed98f9827a9321aa7533c"
server_key_iv: "0f79bc2767507bc84e99a6ec7d1be1cb"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.664733 11984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 14:43:45.684298 11999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:45.686196 11984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 14:43:45.686327 11984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "dc6defe6589e491fab3950f99242ce1b"
format_stamp: "Formatted at 2026-09-29 14:43:45 on dist-test-slave-g8hp"
server_key: "4adb0f05024ed98f9827a9321aa7533c"
server_key_iv: "0f79bc2767507bc84e99a6ec7d1be1cb"
server_key_version: "kuduclusterkey@0"
I20260929 14:43:45.686975 11984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:43:45.717276 11984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:43:45.720842 11984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:43:45.721194 11984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:43:45.721925 11984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:43:45.722946 11984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:43:45.722996 11984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:45.723038 11984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:43:45.723060 11984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:43:45.733582 11984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43809
I20260929 14:43:45.734843 11984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:43:45.741031 12112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43809 every 8 connection(s)
I20260929 14:43:45.742857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 11984
I20260929 14:43:45.742975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:43:45.743278 392 external_mini_cluster.cc:1428] Setting key 60f1252f2864f3a5b20d8318308d7916
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:43:45.750165 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.737081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40613 (local address 127.0.98.62:36735)
0929 14:43:45.738591 (+ 1510us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.738595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.738613 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:43:45.739450 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.739454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.739511 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.739582 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:43:45.740718 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.741382 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.742508 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.742745 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.742833 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.745750 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:43:45.745754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:43:45.745757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:43:45.745812 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:43:45.747607 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.748115 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.748118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.748130 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.748233 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.748633 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.748636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.748638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.748931 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:43:45.749027 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.749931 (+ 904us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.750042 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1420}
I20260929 14:43:45.751381 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.737547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36735 (local address 127.0.98.3:40613)
0929 14:43:45.738129 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.738195 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.739139 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.739754 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.739761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.740056 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:43:45.740563 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.740577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.741499 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.741502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.742393 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.742402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.742889 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.742892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.743056 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.743653 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:43:45.743675 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:43:45.745538 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:43:45.747736 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:45.747741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:45.747756 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:45.748017 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:45.748357 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:43:45.748360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:43:45.748362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:43:45.748540 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:43:45.749097 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:43:45.749105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:43:45.749609 (+ 504us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.749831 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.750231 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":156,"threads_started":1}
I20260929 14:43:45.752625 12113 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36735
I20260929 14:43:45.752938 12113 heartbeater.cc:464] Registering TS with master...
I20260929 14:43:45.753571 12113 heartbeater.cc:511] Master 127.0.98.62:36735 requested a full tablet report, sending...
I20260929 14:43:45.757228 11629 ts_manager.cc:194] Registered new tserver with Master: dc6defe6589e491fab3950f99242ce1b (127.0.98.3:43809)
I20260929 14:43:45.758014 11629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40613
I20260929 14:43:45.758554 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:43:45.770509 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.760994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.62:36735)
0929 14:43:45.761139 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.761144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.761533 (+ 389us) server_negotiation.cc:407] Connection header received
0929 14:43:45.761653 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.761657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.761708 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.761795 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:43:45.762652 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.763446 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.764655 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.765016 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.765125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.766797 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:43:45.766801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:43:45.766804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:43:45.766833 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:43:45.768711 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.769225 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.769228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.769230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.769320 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:43:45.769604 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:43:45.769606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:43:45.769608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:43:45.769848 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:43:45.769968 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.770231 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.770346 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:43:45.774434 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:43:45.778769 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:43:45.797508 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.789612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40135 (local address 127.0.0.1:34938)
0929 14:43:45.790019 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.790414 (+ 395us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.790539 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.792210 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.792225 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.792257 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:43:45.792397 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.792404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.794145 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.794149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.795577 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.795588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.796791 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.796795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.796956 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.797002 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.797120 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.797303 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
I20260929 14:43:45.798055 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.790690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.2:40135)
0929 14:43:45.791047 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.791062 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.791088 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:43:45.791165 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.791340 (+ 175us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.792014 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.792213 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:43:45.792527 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.793873 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.795755 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.796642 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.796768 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.796984 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.797168 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.797279 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":118,"threads_started":1}
I20260929 14:43:45.801354 11913 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a (DEFAULT_TABLE table=test-table [id=7be52925e81d444e8b2fcf26e420d9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:43:45.802611 11913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ab9492bd2480189ed1ec26b74e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:43:45.805102 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.790414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41735 (local address 127.0.0.1:54916)
0929 14:43:45.797777 (+ 7363us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.797790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.797857 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.799445 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.799448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.799465 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:43:45.799554 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.799560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.801004 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.801008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.803184 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.803209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.804609 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.804613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.804724 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.804751 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.804830 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.804950 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7311,"thread_start_us":87,"threads_started":1}
I20260929 14:43:45.805552 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.790562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.98.1:41735)
0929 14:43:45.798986 (+ 8424us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.798992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.799017 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:43:45.799083 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.799087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.799326 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.799468 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:43:45.799672 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.800809 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.803354 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.804255 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.804370 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.804538 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.805303 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.805417 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7413,"thread_start_us":76,"threads_started":1}
I20260929 14:43:45.808651 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a (DEFAULT_TABLE table=test-table [id=7be52925e81d444e8b2fcf26e420d9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:43:45.810896 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ab9492bd2480189ed1ec26b74e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:43:45.812355 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.790210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43809 (local address 127.0.0.1:46576)
0929 14:43:45.797089 (+ 6879us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.797104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.797209 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.806035 (+ 8826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.806039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.806059 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:43:45.806151 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.806156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.807658 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.807662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.808778 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.808789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.811855 (+ 3066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.811859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.811964 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.811992 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.812097 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.812183 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6753,"thread_start_us":76,"threads_started":1}
I20260929 14:43:45.813009 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.790378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46576 (local address 127.0.98.3:43809)
0929 14:43:45.805506 (+ 15128us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.805519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.805542 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:43:45.805619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.805624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.805868 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.806312 (+ 444us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:43:45.806463 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.807526 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.808973 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.809732 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.811831 (+ 2099us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.811990 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.812734 (+ 744us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.812842 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15014,"thread_start_us":122,"threads_started":1}
I20260929 14:43:45.816448 12047 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a (DEFAULT_TABLE table=test-table [id=7be52925e81d444e8b2fcf26e420d9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:43:45.818372 12047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ab9492bd2480189ed1ec26b74e22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:43:45.821347 12131 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Bootstrap starting.
I20260929 14:43:45.823819 12131 tablet_bootstrap.cc:654] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Neither blocks nor log segments found. Creating new log.
I20260929 14:43:45.825304 12131 log.cc:824] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Log is configured to *not* fsync() on all Append() calls
I20260929 14:43:45.826601 12134 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Bootstrap starting.
I20260929 14:43:45.829031 12131 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: No bootstrap required, opened a new log
I20260929 14:43:45.829247 12131 ts_tablet_manager.cc:1397] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:43:45.832971 12132 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Bootstrap starting.
I20260929 14:43:45.832911 12131 raft_consensus.cc:348] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.833230 12131 raft_consensus.cc:374] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:43:45.833294 12131 raft_consensus.cc:729] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3170618b4fb24a798d45caf22db4abbe, State: Initialized, Role: FOLLOWER
I20260929 14:43:45.833832 12134 tablet_bootstrap.cc:654] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Neither blocks nor log segments found. Creating new log.
I20260929 14:43:45.833909 12131 consensus_queue.cc:260] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.835021 12131 ts_tablet_manager.cc:1428] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:43:45.835222 12134 log.cc:824] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:43:45.835408 11979 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
I20260929 14:43:45.835865 12132 tablet_bootstrap.cc:654] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Neither blocks nor log segments found. Creating new log.
I20260929 14:43:45.836673 12132 log.cc:824] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:43:45.837934 12134 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: No bootstrap required, opened a new log
I20260929 14:43:45.838156 12134 ts_tablet_manager.cc:1397] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260929 14:43:45.841939 12134 raft_consensus.cc:348] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.842175 12134 raft_consensus.cc:374] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:43:45.842221 12134 raft_consensus.cc:729] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6defe6589e491fab3950f99242ce1b, State: Initialized, Role: FOLLOWER
I20260929 14:43:45.842736 12134 consensus_queue.cc:260] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.844195 12134 ts_tablet_manager.cc:1428] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:43:45.844566 12113 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
I20260929 14:43:45.845171 12132 tablet_bootstrap.cc:492] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: No bootstrap required, opened a new log
I20260929 14:43:45.845449 12132 ts_tablet_manager.cc:1397] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
W20260929 14:43:45.849696 11980 tablet.cc:2369] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:43:45.850590 12132 raft_consensus.cc:348] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.850835 12132 raft_consensus.cc:374] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:43:45.850890 12132 raft_consensus.cc:729] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ecc59c176b413294325123d5f8f06f, State: Initialized, Role: FOLLOWER
I20260929 14:43:45.851336 12132 consensus_queue.cc:260] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.851984 11843 heartbeater.cc:503] Master 127.0.98.62:36735 was elected leader, sending a full tablet report...
I20260929 14:43:45.853585 12132 ts_tablet_manager.cc:1428] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
Sep 29 14:43:45 dist-test-slave-g8hp krb5kdc[11153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 14:43:45.967226 11844 tablet.cc:2369] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:43:45.986152 12139 raft_consensus.cc:482] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:43:45.986451 12139 raft_consensus.cc:504] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
W20260929 14:43:45.987102 12114 tablet.cc:2369] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:43:45.988083 12139 leader_election.cc:288] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3170618b4fb24a798d45caf22db4abbe (127.0.98.2:40135), dc6defe6589e491fab3950f99242ce1b (127.0.98.3:43809)
I20260929 14:43:45.993707 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.988792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46493 (local address 127.0.98.3:43809)
0929 14:43:45.989014 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.989018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.989033 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:43:45.989087 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.989089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.989145 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.989224 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:43:45.989555 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.990353 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.992278 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.992927 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.993006 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.993097 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.993474 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.993563 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260929 14:43:45.993831 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.988674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44719 (local address 127.0.98.2:40135)
0929 14:43:45.988974 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:45.988979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:45.989749 (+ 770us) server_negotiation.cc:407] Connection header received
0929 14:43:45.989956 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:45.989959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:45.990005 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:45.990048 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:43:45.990438 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.991093 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.992548 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.993142 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.993219 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.993280 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:45.993627 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 14:43:45.993709 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 14:43:45.994248 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.988241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43809 (local address 127.0.98.1:46493)
0929 14:43:45.988633 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.988650 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.988757 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.989269 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.989273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.989298 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:43:45.989413 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.989420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.990516 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.990519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.992128 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.992139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.993234 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.993237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.993338 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.993361 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.993991 (+ 630us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.994069 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":48,"thread_start_us":129,"threads_started":1}
I20260929 14:43:45.994258 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:45.988245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40135 (local address 127.0.98.1:44719)
0929 14:43:45.989658 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:45.989672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:43:45.989831 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:43:45.990179 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:43:45.990182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:43:45.990198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:43:45.990298 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.990304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.991228 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.991231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.992406 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:43:45.992416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:43:45.993386 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:43:45.993389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:43:45.993481 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:45.993502 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:43:45.993968 (+ 466us) client_negotiation.cc:240] Negotiation successful
0929 14:43:45.994054 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"mutex_wait_us":1155,"thread_start_us":86,"threads_started":1}
I20260929 14:43:45.994907 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ab9492bd2480189ed1ec26b74e22a" candidate_uuid: "f8ecc59c176b413294325123d5f8f06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6defe6589e491fab3950f99242ce1b" is_pre_election: true
I20260929 14:43:45.994930 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ab9492bd2480189ed1ec26b74e22a" candidate_uuid: "f8ecc59c176b413294325123d5f8f06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3170618b4fb24a798d45caf22db4abbe" is_pre_election: true
I20260929 14:43:45.995198 12067 raft_consensus.cc:2454] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ecc59c176b413294325123d5f8f06f in term 0.
I20260929 14:43:45.995224 11933 raft_consensus.cc:2454] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ecc59c176b413294325123d5f8f06f in term 0.
I20260929 14:43:45.995693 11731 leader_election.cc:302] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc6defe6589e491fab3950f99242ce1b, f8ecc59c176b413294325123d5f8f06f; no voters:
I20260929 14:43:45.996201 12139 raft_consensus.cc:2781] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:43:45.996304 12139 raft_consensus.cc:482] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:43:45.996359 12139 raft_consensus.cc:3037] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:43:45.997782 12139 raft_consensus.cc:504] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:45.998189 12139 leader_election.cc:288] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [CANDIDATE]: Term 1 election: Requested vote from peers 3170618b4fb24a798d45caf22db4abbe (127.0.98.2:40135), dc6defe6589e491fab3950f99242ce1b (127.0.98.3:43809)
I20260929 14:43:45.998539 11933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ab9492bd2480189ed1ec26b74e22a" candidate_uuid: "f8ecc59c176b413294325123d5f8f06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3170618b4fb24a798d45caf22db4abbe"
I20260929 14:43:45.998662 12067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ab9492bd2480189ed1ec26b74e22a" candidate_uuid: "f8ecc59c176b413294325123d5f8f06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc6defe6589e491fab3950f99242ce1b"
I20260929 14:43:45.998678 11933 raft_consensus.cc:3037] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:43:45.998807 12067 raft_consensus.cc:3037] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:43:45.999899 12067 raft_consensus.cc:2454] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ecc59c176b413294325123d5f8f06f in term 1.
I20260929 14:43:45.999917 11933 raft_consensus.cc:2454] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ecc59c176b413294325123d5f8f06f in term 1.
I20260929 14:43:46.000202 11731 leader_election.cc:302] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc6defe6589e491fab3950f99242ce1b, f8ecc59c176b413294325123d5f8f06f; no voters:
I20260929 14:43:46.000501 12139 raft_consensus.cc:2781] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:43:46.000723 12139 raft_consensus.cc:686] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [term 1 LEADER]: Becoming Leader. State: Replica: f8ecc59c176b413294325123d5f8f06f, State: Running, Role: LEADER
I20260929 14:43:46.001194 12139 consensus_queue.cc:237] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } }
I20260929 14:43:46.004794 11628 catalog_manager.cc:5495] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f reported cstate change: term changed from 0 to 1, leader changed from <none> to f8ecc59c176b413294325123d5f8f06f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f8ecc59c176b413294325123d5f8f06f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8ecc59c176b413294325123d5f8f06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 } health_report { overall_health: UNKNOWN } } }
I20260929 14:43:46.040670 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:46.035355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.0.98.1:41735)
0929 14:43:46.035496 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:43:46.035501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:43:46.035518 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:43:46.035615 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:43:46.035618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:43:46.035692 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:43:46.035825 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:43:46.036134 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:46.036937 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:46.038362 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:43:46.038683 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:43:46.038792 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:43:46.039234 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:43:46.039850 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:43:46.039973 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:43:46.040426 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 14:43:46.040496 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:43:46.049156 11933 raft_consensus.cc:1261] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 1 FOLLOWER]: Refusing update from remote peer f8ecc59c176b413294325123d5f8f06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:43:46.049296 12067 raft_consensus.cc:1261] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 1 FOLLOWER]: Refusing update from remote peer f8ecc59c176b413294325123d5f8f06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:43:46.050022 12139 consensus_queue.cc:1035] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc6defe6589e491fab3950f99242ce1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:43:46.050311 12144 consensus_queue.cc:1035] T 1d3ab9492bd2480189ed1ec26b74e22a P f8ecc59c176b413294325123d5f8f06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3170618b4fb24a798d45caf22db4abbe" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:43:46.066732 12149 mvcc.cc:205] Tried to move back new op lower bound from 7334678634688303104 to 7334678634504122368. Current Snapshot: MvccSnapshot[applied={T|T < 7334678634688303104}]
I20260929 14:43:46.068210 12147 mvcc.cc:205] Tried to move back new op lower bound from 7334678634688303104 to 7334678634504122368. Current Snapshot: MvccSnapshot[applied={T|T < 7334678634688303104}]
I20260929 14:43:46.079921 12148 mvcc.cc:205] Tried to move back new op lower bound from 7334678634688303104 to 7334678634504122368. Current Snapshot: MvccSnapshot[applied={T|T < 7334678634688303104}]
I20260929 14:43:46.090106 11629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:43:46.090313 11629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:43:46.093478 11629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 136ba8a7867c493eb73606e88865e26c: Sending DeleteTablet for 3 replicas of tablet 1d3ab9492bd2480189ed1ec26b74e22a
I20260929 14:43:46.094606 12047 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46576
I20260929 14:43:46.098064 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54916
I20260929 14:43:46.097330 11913 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3ab9492bd2480189ed1ec26b74e22a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34938
I20260929 14:43:46.099407 12156 tablet_replica.cc:330] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: stopping tablet replica
I20260929 14:43:46.099656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 11710
I20260929 14:43:46.101852 12156 raft_consensus.cc:2229] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:43:46.102272 12156 raft_consensus.cc:2258] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:43:46.102717 12158 tablet_replica.cc:330] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: stopping tablet replica
I20260929 14:43:46.103061 12158 raft_consensus.cc:2229] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:43:46.103400 12158 raft_consensus.cc:2258] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:43:46.104741 12156 ts_tablet_manager.cc:1905] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:43:46.104741 12158 ts_tablet_manager.cc:1905] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:43:46.107610 12156 ts_tablet_manager.cc:1918] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:43:46.107811 12156 log.cc:1196] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/1d3ab9492bd2480189ed1ec26b74e22a
I20260929 14:43:46.108219 12156 ts_tablet_manager.cc:1939] T 1d3ab9492bd2480189ed1ec26b74e22a P dc6defe6589e491fab3950f99242ce1b: Deleting consensus metadata
I20260929 14:43:46.109793 11616 catalog_manager.cc:4841] TS dc6defe6589e491fab3950f99242ce1b (127.0.98.3:43809): tablet 1d3ab9492bd2480189ed1ec26b74e22a (table test-table [id=7be52925e81d444e8b2fcf26e420d9f3]) successfully deleted
I20260929 14:43:46.112154 12158 ts_tablet_manager.cc:1918] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:43:46.112289 12158 log.cc:1196] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/1d3ab9492bd2480189ed1ec26b74e22a
I20260929 14:43:46.112627 12158 ts_tablet_manager.cc:1939] T 1d3ab9492bd2480189ed1ec26b74e22a P 3170618b4fb24a798d45caf22db4abbe: Deleting consensus metadata
I20260929 14:43:46.114172 11614 catalog_manager.cc:4841] TS 3170618b4fb24a798d45caf22db4abbe (127.0.98.2:40135): tablet 1d3ab9492bd2480189ed1ec26b74e22a (table test-table [id=7be52925e81d444e8b2fcf26e420d9f3]) successfully deleted
W20260929 14:43:46.115424 11614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41735 (error 108)
I20260929 14:43:46.116513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 11849
I20260929 14:43:46.121901 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:43:46.121593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41735 (local address 127.0.0.1:54930)
0929 14:43:46.121730 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 14:43:46.121804 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 14:43:46.122573 11614 catalog_manager.cc:4584] TS f8ecc59c176b413294325123d5f8f06f (127.0.98.1:41735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d3ab9492bd2480189ed1ec26b74e22a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41735: connect: Connection refused (error 111)
I20260929 14:43:46.123037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 11984
I20260929 14:43:46.129582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 11597
I20260929 14:43:46.135203 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:43:46.470711 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:43:46.470842 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:43:46.808849 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:43:46.809 UTC [11183] LOG: received smart shutdown request
2026-09-29 14:43:46.812 UTC [11183] LOG: background worker "logical replication launcher" (PID 11192) exited with exit code 1
2026-09-29 14:43:46.813 UTC [11187] LOG: shutting down
2026-09-29 14:43:47.027 UTC [11183] LOG: database system is shut down
2026-09-29T14:43:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47292 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:43:47 dist-test-slave-g8hp krb5kdc[12168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:43:47 dist-test-slave-g8hp krb5kdc[12168](info): set up 2 sockets
Sep 29 14:43:47 dist-test-slave-g8hp krb5kdc[12168](info): commencing operation
krb5kdc: starting...
W20260929 14:43:48.092854 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:43:48 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:43:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 14:43:48.264654 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/postgres -l logfile start
2026-09-29 14:43:51.029 UTC [12198] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 14:43:51.029 UTC [12198] LOG: listening on IPv4 address "127.0.98.20", port 50927
2026-09-29 14:43:51.033 UTC [12198] LOG: listening on Unix socket "/tmp/.s.PGSQL.50927"
2026-09-29 14:43:51.062 UTC [12201] LOG: database system was shut down at 2026-09-29 14:43:49 UTC
2026-09-29 14:43:51.067 UTC [12198] LOG: database system is ready to accept connections
I20260929 14:43:51.936532 392 mini_postgres.cc:96] Postgres bound to 50927
2026-09-29 14:43:51.942 UTC [12209] FATAL: database "slave" does not exist
127.0.98.20:50927 - accepting connections
I20260929 14:43:51.943125 392 mini_ranger.cc:161] Starting Ranger...
I20260929 14:43:51.958063 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 14:43:52.528431 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 14:43:52.528501 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin
2026-09-29 14:43:52,789 [I] DB FLAVOR :POSTGRES
2026-09-29 14:43:52,789 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:43:52,789 [I] Checking connection..
2026-09-29 14:43:52,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 14:43:53,041 [I] Checking connection passed.
2026-09-29 14:43:53,041 [I] --------- Verifying version history table ---------
2026-09-29 14:43:53,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:53,291 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:43:53,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:53,555 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 14:43:53,556 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 14:43:53,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 14:43:53.791 UTC [12273] WARNING: there is no transaction in progress
2026-09-29 14:43:53,797 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 14:43:53,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 14:43:54,056 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 14:43:54,057 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:43:54,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 14:43:54,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 14:43:54,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g8hp.c.gcp-upstream.internal','N') ;"
2026-09-29 14:43:54,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:43:55,058 [I] Table x_portal_user does not exist in database ranger
2026-09-29 14:43:55,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:43:55,313 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 14:43:55,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:43:55,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:43:55,806 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 14:43:55,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 14:43:56.860 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:56.879 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:56.898 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:56.915 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:56.935 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:56.957 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.023 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.031 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.048 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.060 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.073 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.088 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.100 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.108 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.117 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57.126 UTC [12381] WARNING: there is no transaction in progress
2026-09-29 14:43:57,908 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 14:43:57,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 14:43:58,150 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 14:43:58,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 14:43:58,396 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 14:43:58,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 14:43:58,653 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 14:43:58,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 14:43:58,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 14:43:59,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g8hp.c.gcp-upstream.internal';"
2026-09-29 14:43:59,388 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 14:43:59,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 14:43:59,637 [I] DB_PATCHES have already been applied
I20260929 14:43:59.643086 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasknwEhRJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 14:43:59.643162 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 14:43:59.646746 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46505
Sep 29, 2026 2:43:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:43:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:44:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:44:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:44:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:44:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:44:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 14:44:00.653308 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.710s user 0.000s sys 0.010s
Sep 29, 2026 2:44:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:44:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 2:44:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:44:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:44:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 14:44:21.890385 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 14:44:21 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693061, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 14:44:21.983570 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 14:44:21.993475 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 14:44:23.058718 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 14:44:23.058794 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms
I20260929 14:44:23.058811 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50927
2026-09-29 14:44:23,213 [I] DB FLAVOR :POSTGRES
2026-09-29 14:44:23,214 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 14:44:23,214 [I] Checking connection
2026-09-29 14:44:23,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 14:44:23,454 [I] connection success
2026-09-29 14:44:23,454 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 14:44:23,454 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 14:44:23,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 14:44:23,697 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 14:44:23,698 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 14:44:23,698 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 14:44:23,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 14:44:23,969 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 14:44:24.476334 392 mini_ranger_kms.cc:310] Created kms service
I20260929 14:44:24.673230 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 14:44:24.721057 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 14:44:24 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693061, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:25.134996 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 14:44:25.135246 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasknwEhRJ/build/debug/bin/postgresql.jar:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasknwEhRJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 14:44:25.135277 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 14:44:25.138885 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12561
I20260929 14:44:25.138978 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58335
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:44:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 14:44:27.164111 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.181s user 0.021s sys 0.118s
I20260929 14:44:27.164242 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 14:44:27.164323 392 mini_ranger_kms.cc:401] 127.0.98.20:58335/kms/v1/keys
Sep 29 14:44:27 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 14:44:27 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:44:28 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 14:44:30 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:44:30.496155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46163
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:30.608701 12612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:30.609030 12612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:30.609078 12612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:30.613245 12612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:30.613313 12612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:30.613332 12612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:30.613348 12612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:30.613361 12612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:30.619087 12612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58335/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46163
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:30.620263 12612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:30.621411 12612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:30.630795 12618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:30.631007 12612 server_base.cc:1034] running on GCE node
W20260929 14:44:30.631086 12620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:30.630793 12617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:30.631831 12612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:30.633065 12612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:30.634244 12612 hybrid_clock.cc:648] HybridClock initialized: now 1790693070634224 us; error 35 us; skew 500 ppm
Sep 29 14:44:30 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:30.642511 12612 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:30.643563 12612 webserver.cc:460] Webserver started at http://127.0.98.62:44587/ using document root <none> and password file <none>
I20260929 14:44:30.644125 12612 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:30.644181 12612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:30.644392 12612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:44:30 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:30.692240 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "535ad500188d43ec88fc61fae5fe1e03"
format_stamp: "Formatted at 2026-09-29 14:44:30 on dist-test-slave-g8hp"
server_key: "34e6159f11277277d2858394b2811a21"
server_key_iv: "e28f5a629f382269d4e9721fd6630fc1"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:30.693004 12612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "535ad500188d43ec88fc61fae5fe1e03"
format_stamp: "Formatted at 2026-09-29 14:44:30 on dist-test-slave-g8hp"
server_key: "34e6159f11277277d2858394b2811a21"
server_key_iv: "e28f5a629f382269d4e9721fd6630fc1"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:30.697046 12612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:44:30.711174 12627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:30.712605 12612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:30.712818 12612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "535ad500188d43ec88fc61fae5fe1e03"
format_stamp: "Formatted at 2026-09-29 14:44:30 on dist-test-slave-g8hp"
server_key: "34e6159f11277277d2858394b2811a21"
server_key_iv: "e28f5a629f382269d4e9721fd6630fc1"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:30.713001 12612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:30.725698 12612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:30.728637 12612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:30.728807 12612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:30.736402 12612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46163
I20260929 14:44:30.736404 12679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46163 every 8 connection(s)
I20260929 14:44:30.737452 12612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:30.740370 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:30.743499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 12612
I20260929 14:44:30.743623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:44:30.743921 392 external_mini_cluster.cc:1428] Setting key 1ecc3fb53b0d585df8afa9be98ab300b
I20260929 14:44:30.746824 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Bootstrap starting.
I20260929 14:44:30.749727 12680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:30.750618 12680 log.cc:824] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:30.752902 12680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: No bootstrap required, opened a new log
Sep 29 14:44:30 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:30.757099 12680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:30.757347 12680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:30.757424 12680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535ad500188d43ec88fc61fae5fe1e03, State: Initialized, Role: FOLLOWER
I20260929 14:44:30.757917 12680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:30.758091 12680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:30.758190 12680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:30.758299 12680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:30.759449 12680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:30.759847 12680 leader_election.cc:302] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 535ad500188d43ec88fc61fae5fe1e03; no voters:
I20260929 14:44:30.760244 12680 leader_election.cc:288] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:44:30.760417 12685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:30.760674 12685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 LEADER]: Becoming Leader. State: Replica: 535ad500188d43ec88fc61fae5fe1e03, State: Running, Role: LEADER
I20260929 14:44:30.761147 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:30.745563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.0.98.62:46163)
0929 14:44:30.746075 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:30.746095 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:30.746138 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:44:30.746960 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:30.746987 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:30.747439 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:30.747842 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:30.748814 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:30.750286 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:30.751523 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:30.752171 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:30.752329 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:30.755581 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:30.755590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:30.755611 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:30.755646 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:30.757686 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:30.758221 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:30.758227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:30.758234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:30.758367 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:30.758667 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:30.758671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:30.758672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:30.759199 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:30.759390 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:30.759684 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 14:44:30.760129 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":163,"threads_started":1}
I20260929 14:44:30.761108 12685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:30.761654 12680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:30.762841 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 535ad500188d43ec88fc61fae5fe1e03. Latest consensus state: current_term: 1 leader_uuid: "535ad500188d43ec88fc61fae5fe1e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } } }
I20260929 14:44:30.762955 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:30.763228 12687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "535ad500188d43ec88fc61fae5fe1e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } } }
I20260929 14:44:30.763327 12687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:30.763844 12680 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:44:30.767386 12694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:30.771397 12694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:30.784624 12694 catalog_manager.cc:1347] Generated new cluster ID: 71e993c17bc641c29ef3749f62648474
I20260929 14:44:30.784708 12694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:30.824790 12694 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:44:30.825817 12694 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:30.833940 12694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Generated new TSK 0
I20260929 14:44:30.834925 12694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:44:31.469740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:31.633044 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:31.633333 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:31.633394 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:31.639464 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:31.639551 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:31.639642 12725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:44:31.647651 12725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:31.649112 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:31.650544 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:31.663959 12733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:31.663959 12735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:31.663986 12732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:31.664381 12725 server_base.cc:1034] running on GCE node
I20260929 14:44:31.664994 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:31.665866 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:31.667062 12725 hybrid_clock.cc:648] HybridClock initialized: now 1790693071667016 us; error 57 us; skew 500 ppm
Sep 29 14:44:31 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:31.670759 12725 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:44:31.672013 12725 webserver.cc:460] Webserver started at http://127.0.98.1:37977/ using document root <none> and password file <none>
I20260929 14:44:31.672796 12725 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:31.672916 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:31.673242 12725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:44:31 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:31.690551 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "92c80770d1a94efdb88ebb721c724226"
format_stamp: "Formatted at 2026-09-29 14:44:31 on dist-test-slave-g8hp"
server_key: "46719ddd9efdcbaeb64932996f959af9"
server_key_iv: "94998240da17e8737ae40bdfd21c2ed7"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:31.691269 12725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92c80770d1a94efdb88ebb721c724226"
format_stamp: "Formatted at 2026-09-29 14:44:31 on dist-test-slave-g8hp"
server_key: "46719ddd9efdcbaeb64932996f959af9"
server_key_iv: "94998240da17e8737ae40bdfd21c2ed7"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:31.695811 12725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:44:31.718701 12742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:31.720539 12725 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260929 14:44:31.720701 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "92c80770d1a94efdb88ebb721c724226"
format_stamp: "Formatted at 2026-09-29 14:44:31 on dist-test-slave-g8hp"
server_key: "46719ddd9efdcbaeb64932996f959af9"
server_key_iv: "94998240da17e8737ae40bdfd21c2ed7"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:31.720844 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:31.738508 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:31.742256 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:31.742501 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:31.743291 12725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:31.744431 12725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:31.744534 12725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:31.744668 12725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:31.744748 12725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:31.755270 12725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39805
I20260929 14:44:31.755339 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39805 every 8 connection(s)
I20260929 14:44:31.756311 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 14:44:31 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:31.767890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 12725
I20260929 14:44:31.768033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:44:31.768343 392 external_mini_cluster.cc:1428] Setting key 6c5bb7f7b4d7e1849c6318b345bfb0d3
I20260929 14:44:31.773140 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:31.758316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60423 (local address 127.0.98.62:46163)
0929 14:44:31.758683 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:31.758689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:31.759443 (+ 754us) server_negotiation.cc:407] Connection header received
0929 14:44:31.760594 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:31.760597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:31.760659 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:31.760742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:31.761873 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:31.762642 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:31.764505 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:31.764925 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:31.765010 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:31.768165 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:31.768169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:31.768172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:31.768205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:31.770393 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:31.771062 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:31.771066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:31.771068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:31.771163 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:31.771643 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:31.771647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:31.771648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:31.771899 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:31.771973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:31.772781 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 14:44:31.772950 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":105,"threads_started":1}
I20260929 14:44:31.775043 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:31.758728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.1:60423)
0929 14:44:31.759234 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:31.759279 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:31.760269 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:31.760967 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:31.760974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:31.761235 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:31.761725 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:31.761739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:31.762797 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:31.762802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:31.764328 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:31.764341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:31.765232 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:31.765236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:31.765455 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:31.766174 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:31.766195 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:31.767945 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:31.770549 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:31.770556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:31.770575 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:31.770946 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:31.771291 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:31.771294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:31.771296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:31.771530 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:31.772059 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:31.772066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:31.772474 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 14:44:31.773319 (+ 845us) client_negotiation.cc:240] Negotiation successful
0929 14:44:31.773898 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":104,"threads_started":1}
I20260929 14:44:31.776813 12856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:31.777122 12856 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:31.777918 12856 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:31.779827 12644 ts_manager.cc:194] Registered new tserver with Master: 92c80770d1a94efdb88ebb721c724226 (127.0.98.1:39805)
I20260929 14:44:31.781487 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60423
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:44:31.813314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:44:31 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 14:44:31.977054 12864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:31.977483 12864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:31.977558 12864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:31.983816 12864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:31.983912 12864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:31.984014 12864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:44:31.992304 12864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:31.994040 12864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:31.995486 12864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:32.009249 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.009392 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.009249 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:32.010233 12864 server_base.cc:1034] running on GCE node
I20260929 14:44:32.010753 12864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:32.011658 12864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:32.014979 12864 hybrid_clock.cc:648] HybridClock initialized: now 1790693072014832 us; error 150 us; skew 500 ppm
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:32.019140 12864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:44:32.020435 12864 webserver.cc:460] Webserver started at http://127.0.98.2:33893/ using document root <none> and password file <none>
I20260929 14:44:32.021306 12864 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:32.021466 12864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:32.021795 12864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:32.039152 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "890b2236c9ab41a4bfb8a2f0a7a229be"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "498dd5616935687db075d33a965d2094"
server_key_iv: "03876353d7740a8f26c06f572b73f254"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.039975 12864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "890b2236c9ab41a4bfb8a2f0a7a229be"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "498dd5616935687db075d33a965d2094"
server_key_iv: "03876353d7740a8f26c06f572b73f254"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.045364 12864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:44:32.066941 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.068745 12864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:44:32.069008 12864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "890b2236c9ab41a4bfb8a2f0a7a229be"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "498dd5616935687db075d33a965d2094"
server_key_iv: "03876353d7740a8f26c06f572b73f254"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.069204 12864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:32.125780 12864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:32.129832 12864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:32.130059 12864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:32.130887 12864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:32.132009 12864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:32.132144 12864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.132244 12864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:32.132303 12864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.143888 12864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38609
I20260929 14:44:32.145210 12864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:44:32.145402 12993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38609 every 8 connection(s)
I20260929 14:44:32.151350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 12864
I20260929 14:44:32.151502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:44:32.151837 392 external_mini_cluster.cc:1428] Setting key 63a7ff4b431f42579a5ff910bc770abe
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:32.163250 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:32.147793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.2:36423)
0929 14:44:32.148283 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:32.148340 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:32.149374 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:32.150252 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:32.150262 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:32.150629 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:32.151347 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:32.151364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.152329 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.152332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:32.153631 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:32.153642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.154356 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.154360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:32.154559 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:32.156150 (+ 1591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:32.156168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:32.157760 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:32.159883 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:32.159890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:32.159909 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:32.160179 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:32.160456 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:32.160459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:32.160461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:32.160624 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:32.161124 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:32.161132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:32.161533 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 14:44:32.161743 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 14:44:32.162123 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260929 14:44:32.164039 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:32.148851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36423 (local address 127.0.98.62:46163)
0929 14:44:32.149000 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:32.149005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:32.149021 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:44:32.149674 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:32.149678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:32.149742 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:32.149801 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:32.151564 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.152177 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.153765 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.154079 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.154152 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:32.157996 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:32.158000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:32.158003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:32.158037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:32.159752 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:32.160289 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:32.160293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:32.160295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:32.160372 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:32.160721 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:32.160724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:32.160726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:32.160970 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:32.161060 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:32.163716 (+ 2656us) server_negotiation.cc:299] Negotiation successful
0929 14:44:32.163862 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:44:32.165685 12994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:32.165952 12994 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:32.166604 12994 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:32.167831 12644 ts_manager.cc:194] Registered new tserver with Master: 890b2236c9ab41a4bfb8a2f0a7a229be (127.0.98.2:38609)
I20260929 14:44:32.168644 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:44:32.190872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 14:44:32.335320 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:32.335765 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:32.335850 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:32.342612 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:32.342718 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:32.342809 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:44:32.351688 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:32.353322 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:32.354763 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:32.368280 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.368790 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.369122 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:32.370630 12999 server_base.cc:1034] running on GCE node
I20260929 14:44:32.371189 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:32.372064 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:32.373468 12999 hybrid_clock.cc:648] HybridClock initialized: now 1790693072373393 us; error 93 us; skew 500 ppm
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:32.377298 12999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:44:32.378599 12999 webserver.cc:460] Webserver started at http://127.0.98.3:45855/ using document root <none> and password file <none>
I20260929 14:44:32.379343 12999 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:32.379432 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:32.379733 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:32.403190 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c0536e19ff4475bbebbac68c003bac8"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "e63d249cda7d803057ab87bbf4fc749c"
server_key_iv: "db6f3f9a22c1380f00b94df5caeeb303"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.403995 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c0536e19ff4475bbebbac68c003bac8"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "e63d249cda7d803057ab87bbf4fc749c"
server_key_iv: "db6f3f9a22c1380f00b94df5caeeb303"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.410346 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 14:44:32.433305 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.435200 12999 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260929 14:44:32.435361 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0c0536e19ff4475bbebbac68c003bac8"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "e63d249cda7d803057ab87bbf4fc749c"
server_key_iv: "db6f3f9a22c1380f00b94df5caeeb303"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.435511 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:32.463858 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:32.467535 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:32.467722 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:32.468380 12999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:32.469496 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:32.469592 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.469671 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:32.469717 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.479928 12999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36135
I20260929 14:44:32.479978 13127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36135 every 8 connection(s)
I20260929 14:44:32.481294 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:44:32.488989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 12999
I20260929 14:44:32.489128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:44:32.489437 392 external_mini_cluster.cc:1428] Setting key cc170eb6f057aa1a7d81ad91ded65eb6
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:32.495760 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:32.483869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41847 (local address 127.0.98.62:46163)
0929 14:44:32.485031 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:32.485036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:32.485058 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:44:32.485984 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:32.485987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:32.486043 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:32.486117 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:32.487236 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.487800 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.489115 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.489437 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.489520 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:32.492355 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:32.492358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:32.492360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:32.492385 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:32.493727 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:32.494213 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:32.494216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:32.494218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:32.494292 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:32.494607 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:32.494610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:32.494612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:32.494795 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:32.494877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:32.495469 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 14:44:32.495599 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062}
I20260929 14:44:32.496695 13130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:32.484325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.3:41847)
0929 14:44:32.484849 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:32.484934 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:32.485766 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:32.486306 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:32.486313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:32.486589 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:32.487089 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:32.487103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.487918 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.487921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:32.488957 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:32.488968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:32.489726 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:32.489730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:32.489933 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:32.490622 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:32.490642 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:32.492198 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:32.493837 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:32.493843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:32.493861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:32.494124 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:32.494371 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:32.494375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:32.494377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:32.494522 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:32.494891 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:32.494898 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:32.495225 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 14:44:32.495421 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 14:44:32.495742 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260929 14:44:32.497814 13128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:32.498059 13128 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:32.498540 13128 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:32.499495 12644 ts_manager.cc:194] Registered new tserver with Master: 0c0536e19ff4475bbebbac68c003bac8 (127.0.98.3:36135)
I20260929 14:44:32.500052 12644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41847
I20260929 14:44:32.503262 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:44:32.503986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 12725
I20260929 14:44:32.509475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 12864
I20260929 14:44:32.514917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 12999
I20260929 14:44:32.520690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 12612
I20260929 14:44:32.526288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46163
--webserver_interface=127.0.98.62
--webserver_port=44587
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46163
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:32.642515 13131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:32.642828 13131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:32.642889 13131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:32.647090 13131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:32.647176 13131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:32.647202 13131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:32.647221 13131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:32.647241 13131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:32.653093 13131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46163
--ranger_config_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:32.654242 13131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:32.655323 13131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:32.663825 13137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:32.664043 13131 server_base.cc:1034] running on GCE node
W20260929 14:44:32.663854 13136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.663836 13139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:32.664734 13131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:32.665896 13131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:32.667083 13131 hybrid_clock.cc:648] HybridClock initialized: now 1790693072667056 us; error 39 us; skew 500 ppm
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:32.669951 13131 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:32.670877 13131 webserver.cc:460] Webserver started at http://127.0.98.62:44587/ using document root <none> and password file <none>
I20260929 14:44:32.671393 13131 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:32.671461 13131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:32.737936 13131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:44:32.740248 13147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:32.741710 13131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:44:32.741832 13131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "535ad500188d43ec88fc61fae5fe1e03"
format_stamp: "Formatted at 2026-09-29 14:44:30 on dist-test-slave-g8hp"
server_key: "34e6159f11277277d2858394b2811a21"
server_key_iv: "e28f5a629f382269d4e9721fd6630fc1"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:32.742388 13131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:32.759073 13131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:32.762048 13131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:32.762213 13131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:32.769356 13131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46163
I20260929 14:44:32.769408 13199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46163 every 8 connection(s)
I20260929 14:44:32.770494 13131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:32.775334 13200 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:44:32.777017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13131
I20260929 14:44:32.777428 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Bootstrap starting.
I20260929 14:44:32.777669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39805
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46163
--webserver_port=37977
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:32.794683 13200 log.cc:824] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:32.801496 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:32.801887 13200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Bootstrap complete.
I20260929 14:44:32.805882 13200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:32.806286 13200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535ad500188d43ec88fc61fae5fe1e03, State: Initialized, Role: FOLLOWER
I20260929 14:44:32.806758 13200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:32.806872 13200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:32.806932 13200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:32.807032 13200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:44:32.808938 13200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:32.809260 13200 leader_election.cc:302] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 535ad500188d43ec88fc61fae5fe1e03; no voters:
I20260929 14:44:32.809561 13200 leader_election.cc:288] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:44:32.809666 13204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:44:32.809871 13204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [term 2 LEADER]: Becoming Leader. State: Replica: 535ad500188d43ec88fc61fae5fe1e03, State: Running, Role: LEADER
I20260929 14:44:32.810230 13204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } }
I20260929 14:44:32.810820 13200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:32.811695 13205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "535ad500188d43ec88fc61fae5fe1e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } } }
I20260929 14:44:32.811839 13205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:32.812201 13213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:32.812222 13206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 535ad500188d43ec88fc61fae5fe1e03. Latest consensus state: current_term: 2 leader_uuid: "535ad500188d43ec88fc61fae5fe1e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "535ad500188d43ec88fc61fae5fe1e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46163 } } }
I20260929 14:44:32.812314 13206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:32.815460 13213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:32.815963 13200 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 14:44:32.816157 13213 catalog_manager.cc:1259] Loaded cluster ID: 71e993c17bc641c29ef3749f62648474
I20260929 14:44:32.816195 13213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:32.818079 13213 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:32.818804 13213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Loaded TSK: 0
I20260929 14:44:32.819775 13213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:44:32.921854 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:32.922376 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:32.922465 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:32.928943 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:32.929087 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:32.929206 13202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:44:32.937320 13202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37977
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:32.938663 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:32.940085 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:32.956355 13238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.957401 13237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:32.958035 13240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:32.958108 13202 server_base.cc:1034] running on GCE node
I20260929 14:44:32.958873 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:32.959755 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:32.960950 13202 hybrid_clock.cc:648] HybridClock initialized: now 1790693072960917 us; error 50 us; skew 500 ppm
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:32.964823 13202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:44:32.966193 13202 webserver.cc:460] Webserver started at http://127.0.98.1:37977/ using document root <none> and password file <none>
I20260929 14:44:32.967005 13202 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:32.967133 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:44:32 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:33.001823 13202 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260929 14:44:33.005640 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.007519 13202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:44:33.007704 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "92c80770d1a94efdb88ebb721c724226"
format_stamp: "Formatted at 2026-09-29 14:44:31 on dist-test-slave-g8hp"
server_key: "46719ddd9efdcbaeb64932996f959af9"
server_key_iv: "94998240da17e8737ae40bdfd21c2ed7"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:33.008507 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:33.027199 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:33.031404 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:33.031701 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:33.032533 13202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:33.033740 13202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:33.033872 13202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.034001 13202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:33.034081 13202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.044754 13202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39805
I20260929 14:44:33.045991 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:44:33.044786 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39805 every 8 connection(s)
I20260929 14:44:33.049615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13202
I20260929 14:44:33.050153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38609
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46163
--webserver_port=33893
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:33.076316 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.049241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44803 (local address 127.0.98.62:46163)
0929 14:44:33.049902 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:33.049926 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:33.053470 (+ 3544us) server_negotiation.cc:407] Connection header received
0929 14:44:33.055155 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:33.055176 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:33.055686 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:33.056059 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:33.058258 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.059898 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.061714 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.062477 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.062623 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.065790 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:33.065797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:33.065818 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:33.065856 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:33.068013 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.072761 (+ 4748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.072770 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.072778 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.073018 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.073478 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.073482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.073483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.074021 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:33.074142 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:33.074921 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 14:44:33.075263 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":132,"threads_started":1}
I20260929 14:44:33.077989 13365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.049393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.1:44803)
0929 14:44:33.053244 (+ 3851us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:33.053317 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:33.054552 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:33.056943 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:33.056952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:33.057317 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:33.058033 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.058055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.060087 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.060091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.061558 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.061570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.062883 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.062887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.063077 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.063764 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:33.063786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:33.065464 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:33.068166 (+ 2702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.068173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.068191 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.068563 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.073151 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.073155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.073157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.073371 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.074212 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:33.074219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:33.074566 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 14:44:33.076706 (+ 2140us) client_negotiation.cc:240] Negotiation successful
0929 14:44:33.077081 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3586,"thread_start_us":162,"threads_started":1}
I20260929 14:44:33.080926 13362 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:33.081240 13362 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.081920 13362 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.084580 13362 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:44:33.084733 13362 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.084998 13362 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:33.086007 13362 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.086225 13362 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.087069 13362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:44:33.190332 13366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:33.190613 13366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:33.190673 13366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:33.194931 13366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:33.195006 13366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:33.195087 13366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:44:33.200682 13366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33893
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:33.201911 13366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:33.203279 13366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:33.213270 13371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:33.213344 13372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:33.213254 13374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:33.214625 13366 server_base.cc:1034] running on GCE node
I20260929 14:44:33.215121 13366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:33.215963 13366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:33.217146 13366 hybrid_clock.cc:648] HybridClock initialized: now 1790693073217106 us; error 51 us; skew 500 ppm
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:33.220830 13366 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:44:33.222025 13366 webserver.cc:460] Webserver started at http://127.0.98.2:33893/ using document root <none> and password file <none>
I20260929 14:44:33.222782 13366 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:33.222937 13366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:33.248759 13366 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 14:44:33.252092 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.253814 13366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 14:44:33.253947 13366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "890b2236c9ab41a4bfb8a2f0a7a229be"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "498dd5616935687db075d33a965d2094"
server_key_iv: "03876353d7740a8f26c06f572b73f254"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:33.254662 13366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:33.273871 13366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:33.277663 13366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:33.277956 13366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:33.278721 13366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:33.279862 13366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:33.279930 13366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.279989 13366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:33.280056 13366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 14:44:33.290930 13366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38609
I20260929 14:44:33.292246 13366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:44:33.290959 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38609 every 8 connection(s)
I20260929 14:44:33.298997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13366
I20260929 14:44:33.299494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58335/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36135
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46163
--webserver_port=45855
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:33.311414 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.294382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36975 (local address 127.0.98.62:46163)
0929 14:44:33.294511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:33.294515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:33.295593 (+ 1078us) server_negotiation.cc:407] Connection header received
0929 14:44:33.296704 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:33.296708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:33.296763 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:33.296816 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:33.298313 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.298926 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.303140 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.303383 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.303479 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.306794 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:33.306799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:33.306802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:33.306839 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:33.308579 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.309313 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.309317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.309319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.309413 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.309890 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.309894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.309896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.310155 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:33.310244 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:33.311035 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 14:44:33.311230 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:44:33.313813 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.294921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.2:36975)
0929 14:44:33.295444 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:33.295487 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:33.296442 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:33.297231 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:33.297240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:33.297586 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:33.298170 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.298184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.301724 (+ 3540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.301728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.302985 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.302999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.303546 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.303550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.303755 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.304445 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:33.304467 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:33.306544 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:33.308741 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.308747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.308768 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.309185 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.309550 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.309553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.309555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.309758 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.310315 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:33.310322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:33.310706 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 14:44:33.312305 (+ 1599us) client_negotiation.cc:240] Negotiation successful
0929 14:44:33.312703 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":163,"threads_started":1}
I20260929 14:44:33.315405 13496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:33.315668 13496 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.316256 13496 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.318100 13496 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:44:33.318233 13496 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.318403 13496 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:33.319262 13496 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.319437 13496 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.320183 13496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:44:33.434034 13499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:33.434455 13499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:33.434545 13499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:33.439488 13499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:33.439574 13499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:33.439661 13499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:44:33.446026 13499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58335/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45855
--tserver_master_addrs=127.0.98.62:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:33.447244 13499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:33.448338 13499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:33.460139 13508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:33.460130 13505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:33.461567 13506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:33.462715 13499 server_base.cc:1034] running on GCE node
I20260929 14:44:33.463222 13499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:33.464178 13499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:33.467480 13499 hybrid_clock.cc:648] HybridClock initialized: now 1790693073467433 us; error 64 us; skew 500 ppm
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:33.470486 13499 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:44:33.471489 13499 webserver.cc:460] Webserver started at http://127.0.98.3:45855/ using document root <none> and password file <none>
I20260929 14:44:33.472163 13499 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:33.472251 13499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:33.489785 13499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:33.492489 13516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.494264 13499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 14:44:33.494414 13499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0c0536e19ff4475bbebbac68c003bac8"
format_stamp: "Formatted at 2026-09-29 14:44:32 on dist-test-slave-g8hp"
server_key: "e63d249cda7d803057ab87bbf4fc749c"
server_key_iv: "db6f3f9a22c1380f00b94df5caeeb303"
server_key_version: "kuduclusterkey@0"
I20260929 14:44:33.495196 13499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:33.519311 13499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:33.522776 13499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:33.523047 13499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:33.523748 13499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:33.524801 13499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:33.524891 13499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.524957 13499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:33.524986 13499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:33.535221 13499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36135
I20260929 14:44:33.535758 13629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36135 every 8 connection(s)
I20260929 14:44:33.536414 13499 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:44:33.539106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13499
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:33.551453 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.539851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.0.98.62:46163)
0929 14:44:33.540043 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:33.540053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:33.541323 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 14:44:33.541374 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:33.541377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:33.541432 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:33.541505 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:33.542497 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.543288 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.544438 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.544956 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.545047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.547078 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:33.547082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:33.547085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:33.547111 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:33.548826 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.549241 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.549245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.549247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.549326 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.549639 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.549642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.549644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.549865 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:33.550075 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:33.551125 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0929 14:44:33.551253 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"spinlock_wait_cycles":3456,"thread_start_us":89,"threads_started":1}
I20260929 14:44:33.554735 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.539760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52673 (local address 127.0.98.62:46163)
0929 14:44:33.539887 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:33.539891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:33.540389 (+ 498us) server_negotiation.cc:407] Connection header received
0929 14:44:33.541472 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:33.541477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:33.541532 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:33.541623 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:33.542991 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.543735 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.545462 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.545795 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.545891 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.550046 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:33.550050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:33.550053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:33.550080 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:33.551697 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.552637 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.552641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.552643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.552735 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:33.553305 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:33.553308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:33.553310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:33.553528 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:33.553609 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:33.554429 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 14:44:33.554559 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:44:33.556510 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:33.538753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46163 (local address 127.0.98.3:52673)
0929 14:44:33.540250 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:33.540293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:33.541238 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:33.541838 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:33.541847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:33.542176 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:33.542836 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.542855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.543880 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.543883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.545336 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:33.545346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:33.546115 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:33.546119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:33.546329 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:33.548217 (+ 1888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:33.548240 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:33.549874 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:33.551828 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.551835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.551854 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.552524 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.552848 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:33.552851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:33.552853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:33.553191 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:33.553678 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:33.553685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:33.554065 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 14:44:33.554907 (+ 842us) client_negotiation.cc:240] Negotiation successful
0929 14:44:33.555322 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":130,"threads_started":1}
I20260929 14:44:33.557927 13630 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46163
I20260929 14:44:33.558178 13630 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.558702 13630 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.560285 13630 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:44:33.560402 13630 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.560571 13630 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:33.561360 13630 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:33.561523 13630 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
W20260929 14:44:33.562232 13630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 14:44:33 dist-test-slave-g8hp krb5kdc[12168](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:34.087401 13362 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:34.087742 13362 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:34.089165 13163 ts_manager.cc:194] Registered new tserver with Master: 92c80770d1a94efdb88ebb721c724226 (127.0.98.1:39805)
I20260929 14:44:34.090968 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44803
I20260929 14:44:34.320415 13496 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:34.320734 13496 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:34.321666 13163 ts_manager.cc:194] Registered new tserver with Master: 890b2236c9ab41a4bfb8a2f0a7a229be (127.0.98.2:38609)
I20260929 14:44:34.322454 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36975
Sep 29 14:44:34 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 14:44:34.562506 13630 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:34.562878 13630 heartbeater.cc:511] Master 127.0.98.62:46163 requested a full tablet report, sending...
I20260929 14:44:34.563931 13163 ts_manager.cc:194] Registered new tserver with Master: 0c0536e19ff4475bbebbac68c003bac8 (127.0.98.3:36135)
I20260929 14:44:34.564754 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52673
I20260929 14:44:34.568988 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:44:34.580977 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.570807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.98.62:46163)
0929 14:44:34.571380 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:34.571384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:34.571706 (+ 322us) server_negotiation.cc:407] Connection header received
0929 14:44:34.571750 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:34.571753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:34.571799 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:34.571867 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:34.572762 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.573572 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.574718 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.575053 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.575136 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.576928 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:34.576932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:34.576935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:34.576966 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:34.578622 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:34.579107 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:34.579111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:34.579113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:34.579198 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:34.579467 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:34.579471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:34.579472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:34.579695 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:34.579828 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:34.580524 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 14:44:34.580743 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":91,"threads_started":1}
I20260929 14:44:34.586153 13163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:44:34.590998 13163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:44:34.620256 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.610693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36135 (local address 127.0.0.1:49664)
0929 14:44:34.613507 (+ 2814us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:34.613565 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:34.613696 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:34.615232 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:34.615239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:34.615263 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:34.615408 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.615416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.616903 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.616906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.618325 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.618337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.619664 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.619668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.619790 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.619841 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 14:44:34.619949 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:44:34.620086 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"thread_start_us":116,"threads_started":1}
I20260929 14:44:34.620940 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.611529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.0.98.3:36135)
0929 14:44:34.614517 (+ 2988us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:34.614524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:34.614547 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:44:34.614620 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:34.614625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:34.614859 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:34.615022 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:34.615545 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.616739 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.618470 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.619349 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.619450 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.619603 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:34.620652 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0929 14:44:34.620766 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2876,"thread_start_us":83,"threads_started":1}
I20260929 14:44:34.624285 13564 tablet_service.cc:1467] Processing CreateTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c (DEFAULT_TABLE table=test-table [id=dcf84ee7fb3349579b43a66cd3378147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:34.625517 13564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bbe64662d24fdca14cd9e2a6eab95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:34.628268 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.612312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38609 (local address 127.0.0.1:54590)
0929 14:44:34.613929 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:34.613941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:34.614018 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:34.623976 (+ 9958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:34.623980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:34.624004 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:34.624110 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.624116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.625680 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.625684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.626805 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.626818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.627861 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.627865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.627959 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.627988 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:44:34.628032 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 14:44:34.628109 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":96,"threads_started":1}
I20260929 14:44:34.628924 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.612459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.0.98.2:38609)
0929 14:44:34.623469 (+ 11010us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:34.623476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:34.623504 (+ 28us) server_negotiation.cc:407] Connection header received
0929 14:44:34.623584 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:34.623590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:34.623798 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:34.624261 (+ 463us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:34.624433 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.625543 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.626964 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.627720 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.628479 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.628627 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:34.628676 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 14:44:34.628766 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10887,"thread_start_us":60,"threads_started":1}
I20260929 14:44:34.632094 13430 tablet_service.cc:1467] Processing CreateTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c (DEFAULT_TABLE table=test-table [id=dcf84ee7fb3349579b43a66cd3378147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:34.633312 13430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bbe64662d24fdca14cd9e2a6eab95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:34.640992 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.613036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.0.1:49426)
0929 14:44:34.614181 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:34.614193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:34.614267 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:34.636299 (+ 22032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:34.636303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:34.636329 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:34.636447 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.636453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.638016 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.638020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.639233 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:34.639246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.640504 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.640509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:34.640607 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.640638 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 14:44:34.640695 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:44:34.640790 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"thread_start_us":85,"threads_started":1}
I20260929 14:44:34.641757 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:34.613190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.0.98.1:39805)
0929 14:44:34.635776 (+ 22586us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:34.635784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:34.635824 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:44:34.635912 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:34.635917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:34.636128 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:34.636610 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:34.636764 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.637868 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.639394 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:34.640269 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:34.641305 (+ 1036us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:34.641470 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:34.641524 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:44:34.641620 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22463,"thread_start_us":115,"threads_started":1}
I20260929 14:44:34.645148 13296 tablet_service.cc:1467] Processing CreateTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c (DEFAULT_TABLE table=test-table [id=dcf84ee7fb3349579b43a66cd3378147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:34.646407 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bbe64662d24fdca14cd9e2a6eab95c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:34.656733 13663 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Bootstrap starting.
I20260929 14:44:34.659384 13663 tablet_bootstrap.cc:654] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:34.660389 13663 log.cc:824] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:34.661262 13665 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Bootstrap starting.
I20260929 14:44:34.664407 13665 tablet_bootstrap.cc:654] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:34.665405 13664 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Bootstrap starting.
I20260929 14:44:34.668033 13664 tablet_bootstrap.cc:654] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:34.669998 13665 log.cc:824] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:34.672091 13663 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: No bootstrap required, opened a new log
I20260929 14:44:34.672333 13663 ts_tablet_manager.cc:1397] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260929 14:44:34.677114 13664 log.cc:824] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:34.677892 13663 raft_consensus.cc:348] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.678225 13663 raft_consensus.cc:374] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:34.678289 13663 raft_consensus.cc:729] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0536e19ff4475bbebbac68c003bac8, State: Initialized, Role: FOLLOWER
I20260929 14:44:34.678886 13663 consensus_queue.cc:260] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.680094 13663 ts_tablet_manager.cc:1428] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 14:44:34.682618 13665 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: No bootstrap required, opened a new log
I20260929 14:44:34.682889 13665 ts_tablet_manager.cc:1397] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260929 14:44:34.687463 13665 raft_consensus.cc:348] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.687827 13665 raft_consensus.cc:374] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:34.687914 13665 raft_consensus.cc:729] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c80770d1a94efdb88ebb721c724226, State: Initialized, Role: FOLLOWER
I20260929 14:44:34.688427 13630 heartbeater.cc:503] Master 127.0.98.62:46163 was elected leader, sending a full tablet report...
I20260929 14:44:34.692152 13665 consensus_queue.cc:260] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.693791 13665 ts_tablet_manager.cc:1428] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260929 14:44:34.694154 13362 heartbeater.cc:503] Master 127.0.98.62:46163 was elected leader, sending a full tablet report...
I20260929 14:44:34.695853 13664 tablet_bootstrap.cc:492] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: No bootstrap required, opened a new log
I20260929 14:44:34.696077 13664 ts_tablet_manager.cc:1397] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.004s
I20260929 14:44:34.701205 13664 raft_consensus.cc:348] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.701551 13664 raft_consensus.cc:374] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:34.701599 13664 raft_consensus.cc:729] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890b2236c9ab41a4bfb8a2f0a7a229be, State: Initialized, Role: FOLLOWER
I20260929 14:44:34.702188 13664 consensus_queue.cc:260] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:34.703339 13664 ts_tablet_manager.cc:1428] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:44:34.703704 13496 heartbeater.cc:503] Master 127.0.98.62:46163 was elected leader, sending a full tablet report...
Sep 29 14:44:34 dist-test-slave-g8hp krb5kdc[12168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260929 14:44:34.789100 13631 tablet.cc:2369] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:44:34.794552 13497 tablet.cc:2369] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:44:34.799074 13363 tablet.cc:2369] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:44:35.042034 13672 raft_consensus.cc:482] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:44:35.042318 13672 raft_consensus.cc:504] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:35.043808 13672 leader_election.cc:288] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c0536e19ff4475bbebbac68c003bac8 (127.0.98.3:36135), 92c80770d1a94efdb88ebb721c724226 (127.0.98.1:39805)
I20260929 14:44:35.048960 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.044157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45783 (local address 127.0.98.3:36135)
0929 14:44:35.044310 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:35.044315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:35.044556 (+ 241us) server_negotiation.cc:407] Connection header received
0929 14:44:35.044777 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:35.044781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:35.044834 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:35.044925 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:35.045363 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.046107 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.047637 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.048237 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.048307 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:35.048376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:35.048693 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 14:44:35.048774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:44:35.049335 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.044699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34929 (local address 127.0.98.1:39805)
0929 14:44:35.045171 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:35.045175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:35.045188 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:44:35.045234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:35.045237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:35.045282 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:35.045376 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:35.045803 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.046557 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.048121 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.048691 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.048758 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:35.048813 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:35.049143 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 14:44:35.049226 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260929 14:44:35.049502 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.044108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36135 (local address 127.0.98.2:45783)
0929 14:44:35.044474 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:35.044488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:35.044644 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:35.045080 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:35.045083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:35.045097 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:35.045237 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:35.045244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.046235 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.046238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:35.047502 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:35.047513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.048465 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.048468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:35.048563 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:35.048584 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:44:35.049216 (+ 632us) client_negotiation.cc:240] Negotiation successful
0929 14:44:35.049291 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260929 14:44:35.049697 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.044556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.98.2:34929)
0929 14:44:35.044916 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:35.044928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:35.045006 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:35.045514 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:35.045517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:35.045534 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:35.045640 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:35.045645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.046750 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.046753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:35.047947 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:35.048007 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.048924 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.048927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:35.049013 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:35.049033 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:44:35.049487 (+ 454us) client_negotiation.cc:240] Negotiation successful
0929 14:44:35.049558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":89,"threads_started":1}
I20260929 14:44:35.050324 13583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bbe64662d24fdca14cd9e2a6eab95c" candidate_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0536e19ff4475bbebbac68c003bac8" is_pre_election: true
I20260929 14:44:35.050328 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bbe64662d24fdca14cd9e2a6eab95c" candidate_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c80770d1a94efdb88ebb721c724226" is_pre_election: true
I20260929 14:44:35.050652 13583 raft_consensus.cc:2454] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890b2236c9ab41a4bfb8a2f0a7a229be in term 0.
I20260929 14:44:35.050652 13316 raft_consensus.cc:2454] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890b2236c9ab41a4bfb8a2f0a7a229be in term 0.
I20260929 14:44:35.051174 13386 leader_election.cc:302] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 890b2236c9ab41a4bfb8a2f0a7a229be, 92c80770d1a94efdb88ebb721c724226; no voters:
I20260929 14:44:35.051504 13672 raft_consensus.cc:2781] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:44:35.051599 13672 raft_consensus.cc:482] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:44:35.051635 13672 raft_consensus.cc:3037] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:35.053033 13672 raft_consensus.cc:504] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:35.053409 13672 leader_election.cc:288] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [CANDIDATE]: Term 1 election: Requested vote from peers 0c0536e19ff4475bbebbac68c003bac8 (127.0.98.3:36135), 92c80770d1a94efdb88ebb721c724226 (127.0.98.1:39805)
I20260929 14:44:35.053807 13583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bbe64662d24fdca14cd9e2a6eab95c" candidate_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c0536e19ff4475bbebbac68c003bac8"
I20260929 14:44:35.053874 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bbe64662d24fdca14cd9e2a6eab95c" candidate_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c80770d1a94efdb88ebb721c724226"
I20260929 14:44:35.053944 13583 raft_consensus.cc:3037] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:35.054005 13316 raft_consensus.cc:3037] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:35.055066 13583 raft_consensus.cc:2454] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890b2236c9ab41a4bfb8a2f0a7a229be in term 1.
I20260929 14:44:35.055245 13316 raft_consensus.cc:2454] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890b2236c9ab41a4bfb8a2f0a7a229be in term 1.
I20260929 14:44:35.055380 13386 leader_election.cc:302] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c0536e19ff4475bbebbac68c003bac8, 890b2236c9ab41a4bfb8a2f0a7a229be; no voters:
I20260929 14:44:35.055575 13672 raft_consensus.cc:2781] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:35.055791 13672 raft_consensus.cc:686] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 1 LEADER]: Becoming Leader. State: Replica: 890b2236c9ab41a4bfb8a2f0a7a229be, State: Running, Role: LEADER
I20260929 14:44:35.056167 13672 consensus_queue.cc:237] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260929 14:44:35.059783 13163 catalog_manager.cc:5495] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be reported cstate change: term changed from 0 to 1, leader changed from <none> to 890b2236c9ab41a4bfb8a2f0a7a229be (127.0.98.2). New cstate: current_term: 1 leader_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "890b2236c9ab41a4bfb8a2f0a7a229be" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } health_report { overall_health: UNKNOWN } } }
I20260929 14:44:35.072543 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.068923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.0.98.2:38609)
0929 14:44:35.069063 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:35.069068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:35.069083 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:44:35.069234 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:35.069237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:35.069295 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:35.069388 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:44:35.069667 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.070232 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.071084 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:35.071308 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:35.071396 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:35.071701 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:44:35.072138 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:44:35.072249 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:35.072319 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:44:35.072380 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:44:35.079385 13316 raft_consensus.cc:1261] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226 [term 1 FOLLOWER]: Refusing update from remote peer 890b2236c9ab41a4bfb8a2f0a7a229be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:35.079385 13583 raft_consensus.cc:1261] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 1 FOLLOWER]: Refusing update from remote peer 890b2236c9ab41a4bfb8a2f0a7a229be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:35.080164 13683 consensus_queue.cc:1035] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c80770d1a94efdb88ebb721c724226" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:35.080392 13672 consensus_queue.cc:1035] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c0536e19ff4475bbebbac68c003bac8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:35.088162 13686 mvcc.cc:205] Tried to move back new op lower bound from 7334678835516940288 to 7334678835432984576. Current Snapshot: MvccSnapshot[applied={T|T < 7334678835516940288}]
I20260929 14:44:35.088749 13688 mvcc.cc:205] Tried to move back new op lower bound from 7334678835516940288 to 7334678835432984576. Current Snapshot: MvccSnapshot[applied={T|T < 7334678835516940288}]
I20260929 14:44:35.094545 13687 mvcc.cc:205] Tried to move back new op lower bound from 7334678835516940288 to 7334678835432984576. Current Snapshot: MvccSnapshot[applied={T|T < 7334678835516940288}]
I20260929 14:44:35.106870 13164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:35.107086 13164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:35.110037 13164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 535ad500188d43ec88fc61fae5fe1e03: Sending DeleteTablet for 3 replicas of tablet f5bbe64662d24fdca14cd9e2a6eab95c
I20260929 14:44:35.110989 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49426
I20260929 14:44:35.111495 13430 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54590
I20260929 14:44:35.111740 13564 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bbe64662d24fdca14cd9e2a6eab95c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49664
I20260929 14:44:35.111974 13697 tablet_replica.cc:330] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: stopping tablet replica
I20260929 14:44:35.112308 13697 raft_consensus.cc:2229] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:44:35.112709 13696 tablet_replica.cc:330] T f5bbe64662d24fdca14cd9e2a6eab95c P 92c80770d1a94efdb88ebb721c724226: stopping tablet replica
I20260929 14:44:35.112805 13697 raft_consensus.cc:2258] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:35.113054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13202
I20260929 14:44:35.115427 13697 ts_tablet_manager.cc:1905] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:35.118467 13697 ts_tablet_manager.cc:1918] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:35.118583 13697 log.cc:1196] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/f5bbe64662d24fdca14cd9e2a6eab95c
I20260929 14:44:35.118896 13697 ts_tablet_manager.cc:1939] T f5bbe64662d24fdca14cd9e2a6eab95c P 890b2236c9ab41a4bfb8a2f0a7a229be: Deleting consensus metadata
I20260929 14:44:35.120293 13148 catalog_manager.cc:4841] TS 890b2236c9ab41a4bfb8a2f0a7a229be (127.0.98.2:38609): tablet f5bbe64662d24fdca14cd9e2a6eab95c (table test-table [id=dcf84ee7fb3349579b43a66cd3378147]) successfully deleted
W20260929 14:44:35.120404 13151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39805 (error 108)
I20260929 14:44:35.120884 13698 tablet_replica.cc:330] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: stopping tablet replica
I20260929 14:44:35.121256 13698 raft_consensus.cc:2229] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:35.121587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13366
I20260929 14:44:35.121630 13698 raft_consensus.cc:2258] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:35.123584 13698 ts_tablet_manager.cc:1905] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:35.125236 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:35.124132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.0.1:49432)
0929 14:44:35.125031 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:35.125128 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":798}
W20260929 14:44:35.125581 13151 catalog_manager.cc:4584] TS 92c80770d1a94efdb88ebb721c724226 (127.0.98.1:39805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5bbe64662d24fdca14cd9e2a6eab95c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39805: connect: Connection refused (error 111)
I20260929 14:44:35.126503 13698 ts_tablet_manager.cc:1918] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:35.126616 13698 log.cc:1196] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/f5bbe64662d24fdca14cd9e2a6eab95c
I20260929 14:44:35.126930 13698 ts_tablet_manager.cc:1939] T f5bbe64662d24fdca14cd9e2a6eab95c P 0c0536e19ff4475bbebbac68c003bac8: Deleting consensus metadata
I20260929 14:44:35.129019 13151 catalog_manager.cc:4841] TS 0c0536e19ff4475bbebbac68c003bac8 (127.0.98.3:36135): tablet f5bbe64662d24fdca14cd9e2a6eab95c (table test-table [id=dcf84ee7fb3349579b43a66cd3378147]) successfully deleted
I20260929 14:44:35.129187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13499
I20260929 14:44:35.134955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13131
I20260929 14:44:35.142866 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 14:44:35.482213 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 14:44:35.482339 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 14:44:35.828461 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 14:44:35.828 UTC [12198] LOG: received smart shutdown request
2026-09-29 14:44:35.833 UTC [12198] LOG: background worker "logical replication launcher" (PID 12207) exited with exit code 1
2026-09-29 14:44:35.834 UTC [12202] LOG: shutting down
2026-09-29 14:44:36.075 UTC [12198] LOG: database system is shut down
2026-09-29T14:44:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49060 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:44:36 dist-test-slave-g8hp krb5kdc[13708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:44:36 dist-test-slave-g8hp krb5kdc[13708](info): set up 2 sockets
Sep 29 14:44:36 dist-test-slave-g8hp krb5kdc[13708](info): commencing operation
krb5kdc: starting...
W20260929 14:44:37.164904 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:44:37.294199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:37.407517 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:37.407816 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:37.407889 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:37.412022 13721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:37.412099 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:37.412127 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:37.412147 13721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:37.412168 13721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:37.417819 13721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:37.418944 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:37.419981 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:37.428669 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.428658 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.428661 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:37.428732 13721 server_base.cc:1034] running on GCE node
I20260929 14:44:37.429507 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:37.430670 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:37.431936 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790693077431816 us; error 38 us; skew 500 ppm
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:37.435117 13721 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:37.436120 13721 webserver.cc:460] Webserver started at http://127.0.98.62:40833/ using document root <none> and password file <none>
I20260929 14:44:37.436682 13721 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:37.436748 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:37.436985 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:37.439210 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "138539603fe242eb90cfa14705930a10"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "d0cce1f8af9d159f3db79e8f18400459"
server_key_iv: "7d1233af04688c750992f1de669e5647"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.439793 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "138539603fe242eb90cfa14705930a10"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "d0cce1f8af9d159f3db79e8f18400459"
server_key_iv: "7d1233af04688c750992f1de669e5647"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.443785 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:44:37.446448 13736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:37.447830 13721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:44:37.448035 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "138539603fe242eb90cfa14705930a10"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "d0cce1f8af9d159f3db79e8f18400459"
server_key_iv: "7d1233af04688c750992f1de669e5647"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.448153 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:37.462119 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:37.465508 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:37.465693 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:37.472820 13721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33697
I20260929 14:44:37.473006 13788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33697 every 8 connection(s)
I20260929 14:44:37.473897 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:37.476789 13789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:37.480625 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.481201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13721
I20260929 14:44:37.481334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:44:37.481622 392 external_mini_cluster.cc:1428] Setting key fae6cbd285b73fb5179db4a5326a2e73
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:37.499172 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.483346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.0.1:53956)
0929 14:44:37.483861 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.484020 (+ 159us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.485055 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.485808 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.485814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.486311 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.486695 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.486718 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.487942 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.487946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.489158 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.489171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.489976 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.489981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.490118 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.490626 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.490642 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.493893 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.496015 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.496025 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.496038 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.496370 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.496738 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.496741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.496743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.496895 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.497704 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.497709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.497846 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.498018 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.498314 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
I20260929 14:44:37.499223 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.483346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.0.98.62:33697)
0929 14:44:37.483887 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.483907 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.484148 (+ 241us) server_negotiation.cc:407] Connection header received
0929 14:44:37.485405 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.485410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.485595 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.485754 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.486878 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.487801 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.489282 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.489739 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.489907 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.494060 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.494068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.494085 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.494113 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.495826 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.496486 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.496502 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.496504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.496651 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.496986 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.496989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.497000 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.497568 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.497707 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.498143 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.498438 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":421,"thread_start_us":137,"threads_started":1}
I20260929 14:44:37.502075 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.502839 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.502567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56036)
0929 14:44:37.502695 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.502773 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 14:44:37.503078 13738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
I20260929 14:44:37.503943 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.503740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56046)
0929 14:44:37.503838 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.503888 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:44:37.504155 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41291: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:44:37.512413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41291
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:37.553444 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } attempt: 1
I20260929 14:44:37.554268 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.553992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56058)
0929 14:44:37.554135 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.554194 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 14:44:37.554953 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.554776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56068)
0929 14:44:37.554859 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.554901 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 14:44:37.555078 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41291: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
I20260929 14:44:37.623250 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } attempt: 2
I20260929 14:44:37.624123 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.623770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56070)
0929 14:44:37.623953 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.624031 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 14:44:37.625077 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.624682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56086)
0929 14:44:37.624958 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.625011 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
W20260929 14:44:37.625244 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41291: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41291: connect: Connection refused (error 111)
W20260929 14:44:37.629602 13796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:37.629881 13796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:37.629945 13796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:37.634013 13796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:37.634083 13796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:37.634101 13796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:37.634116 13796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:37.634131 13796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:37.639720 13796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:37.640841 13796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:37.641913 13796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:37.650596 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.650545 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.650537 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:37.650612 13796 server_base.cc:1034] running on GCE node
I20260929 14:44:37.651430 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:37.652595 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:37.653759 13796 hybrid_clock.cc:648] HybridClock initialized: now 1790693077653726 us; error 48 us; skew 500 ppm
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:37.656586 13796 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:44:37.657488 13796 webserver.cc:460] Webserver started at http://127.0.98.61:35259/ using document root <none> and password file <none>
I20260929 14:44:37.658033 13796 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:37.658116 13796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:37.658305 13796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:37.660332 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data/instance:
uuid: "b28ca853dfa4404fb31c1c073f62d8ba"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "9cb4e9d18f7e1848160b63c0333b61e0"
server_key_iv: "b3f5294d4b80188167df75fe5944c9c0"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.660832 13796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal/instance:
uuid: "b28ca853dfa4404fb31c1c073f62d8ba"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "9cb4e9d18f7e1848160b63c0333b61e0"
server_key_iv: "b3f5294d4b80188167df75fe5944c9c0"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.664597 13796 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:44:37.667201 13811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:37.668674 13796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:37.668807 13796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "b28ca853dfa4404fb31c1c073f62d8ba"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "9cb4e9d18f7e1848160b63c0333b61e0"
server_key_iv: "b3f5294d4b80188167df75fe5944c9c0"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.668970 13796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:37.696606 13796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:37.700387 13796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:37.700611 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:37.707702 13796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41291
I20260929 14:44:37.707721 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41291 every 8 connection(s)
I20260929 14:44:37.708683 13796 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:44:37.709100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13796
I20260929 14:44:37.709235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-1/wal/instance
I20260929 14:44:37.709519 392 external_mini_cluster.cc:1428] Setting key b69ec3fba55432623c2149ea19114bca
I20260929 14:44:37.712049 13864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:37.715911 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 14:44:37.716388 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:37.729557 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.717906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.0.98.62:33697)
0929 14:44:37.718091 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.718096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.718460 (+ 364us) server_negotiation.cc:407] Connection header received
0929 14:44:37.719309 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.719313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.719369 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.719485 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.720981 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.721732 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.723111 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.723480 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.723861 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.726138 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.726142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.726144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.726170 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.727469 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.727897 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.727900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.727902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.727973 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.728279 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.728283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.728285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.728492 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.728571 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.729206 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.729358 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":101}
I20260929 14:44:37.729658 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.716938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56096)
0929 14:44:37.717077 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.717093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.717194 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.719571 (+ 2377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.719574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.719766 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.719900 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.719909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.721734 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.721738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.722296 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.722307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.723246 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.723249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.723373 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.723869 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.723884 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.725220 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.727467 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.727471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.727473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.727712 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.728107 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.728110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.728112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.728185 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.728994 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.728997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.729091 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.729199 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.729448 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":92}
I20260929 14:44:37.730535 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.717863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.0.1:53962)
0929 14:44:37.718317 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.718363 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.719076 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.719564 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.719574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.720007 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.720765 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.720779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.722276 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.722279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.723001 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.723010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.723630 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.723635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.723747 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.724265 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.724286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.726013 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.727567 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.727575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.727586 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.727798 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.728050 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.728053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.728055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.728187 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.728614 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.728620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.728911 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.729143 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.729528 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":45,"spinlock_wait_cycles":10880,"thread_start_us":138,"threads_started":1}
I20260929 14:44:37.730532 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.717680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.0.98.61:41291)
0929 14:44:37.718144 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.718161 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.718205 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:44:37.719066 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.719090 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.719393 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.720070 (+ 677us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.720544 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.721627 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.722426 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.723131 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.723292 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.725379 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.725387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.725411 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.725438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.727311 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.727844 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.727850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.727852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.727986 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.728610 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.728612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.728613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.728902 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.729009 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.729290 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.729601 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":319,"spinlock_wait_cycles":3968,"thread_start_us":150,"threads_started":1}
I20260929 14:44:37.731859 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.732651 13789 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.733412 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.733138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33394)
0929 14:44:37.733243 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.733298 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:44:37.734052 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.733820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33404)
0929 14:44:37.733946 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.733993 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 14:44:37.734221 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 14:44:37.740080 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.732350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56102)
0929 14:44:37.732455 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.732471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.732551 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.732770 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.732773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.732968 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.733065 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.733071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.733911 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.733914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.734542 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.734552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.735024 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.735028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.735116 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.735588 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.735600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.736971 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.738499 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.738503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.738505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.738716 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.739014 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.739017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.739019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.739098 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.739513 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.739515 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.739601 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.739707 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.739858 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 14:44:37.740096 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.732420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.98.61:41291)
0929 14:44:37.732522 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.732526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.732540 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:44:37.732622 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.732625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.732666 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.732742 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.733166 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.733807 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.734642 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.734921 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.735008 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.737117 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.737121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.737122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.737143 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.738374 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.738852 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.738855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.738856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.738936 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.739158 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.739160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.739162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.739356 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.739433 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.739766 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.739943 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":37}
I20260929 14:44:37.740797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44145
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:37.740833 13864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.741588 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.741338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33406)
0929 14:44:37.741452 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.741531 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:44:37.742245 13814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
I20260929 14:44:37.743132 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.742862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33420)
0929 14:44:37.743008 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.743064 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 14:44:37.743278 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
I20260929 14:44:37.765367 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 1
I20260929 14:44:37.766292 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.765951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33428)
0929 14:44:37.766123 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.766197 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 14:44:37.767011 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.766794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33436)
0929 14:44:37.766894 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.766952 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 14:44:37.767158 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
I20260929 14:44:37.792560 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 1
I20260929 14:44:37.793560 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.793207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33452)
0929 14:44:37.793380 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.793467 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 14:44:37.794338 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.794107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33468)
0929 14:44:37.794220 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.794274 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 14:44:37.794484 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
I20260929 14:44:37.842332 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 2
I20260929 14:44:37.843272 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.842930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33484)
0929 14:44:37.843104 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.843178 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 14:44:37.844027 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.843789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33490)
0929 14:44:37.843902 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.843956 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:44:37.844194 13789 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
W20260929 14:44:37.854640 13870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:37.854909 13870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:37.854957 13870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:37.859033 13870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:37.859093 13870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:37.859112 13870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:37.859127 13870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:37.859143 13870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:37.862658 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 2
I20260929 14:44:37.863538 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.863220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33502)
0929 14:44:37.863373 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.863451 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 14:44:37.864239 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.864041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33516)
0929 14:44:37.864134 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.864182 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 14:44:37.864349 13864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44145: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44145: connect: Connection refused (error 111)
I20260929 14:44:37.864964 13870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33697,127.0.98.61:41291,127.0.98.60:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:37.865981 13870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:37.866956 13870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:37.875588 13877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.875599 13876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:37.875599 13879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:37.875735 13870 server_base.cc:1034] running on GCE node
I20260929 14:44:37.876681 13870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:37.877931 13870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:37.879114 13870 hybrid_clock.cc:648] HybridClock initialized: now 1790693077879090 us; error 46 us; skew 500 ppm
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:37.882326 13870 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:44:37.883330 13870 webserver.cc:460] Webserver started at http://127.0.98.60:37721/ using document root <none> and password file <none>
I20260929 14:44:37.883965 13870 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:37.884038 13870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:37.884253 13870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:37.886531 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data/instance:
uuid: "4b713b6d5127427a9dac8d8d43e636cf"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "417b6c0fd611b52c50334a7947be710e"
server_key_iv: "14a048f5dddbff8c21e6fab9087164e4"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.887116 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal/instance:
uuid: "4b713b6d5127427a9dac8d8d43e636cf"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "417b6c0fd611b52c50334a7947be710e"
server_key_iv: "14a048f5dddbff8c21e6fab9087164e4"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.891433 13870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:44:37.894268 13886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:37.895790 13870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:44:37.895927 13870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "4b713b6d5127427a9dac8d8d43e636cf"
format_stamp: "Formatted at 2026-09-29 14:44:37 on dist-test-slave-g8hp"
server_key: "417b6c0fd611b52c50334a7947be710e"
server_key_iv: "14a048f5dddbff8c21e6fab9087164e4"
server_key_version: "encryptionkey@0"
I20260929 14:44:37.896036 13870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:37.909153 13870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:37.912330 13870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:37.912500 13870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:37.919632 13870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44145
I20260929 14:44:37.919653 13938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44145 every 8 connection(s)
I20260929 14:44:37.920626 13870 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:44:37.923511 13939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:37.927546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 13870
I20260929 14:44:37.927691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790692915364895-392-0/minicluster-data/master-2/wal/instance
I20260929 14:44:37.927366 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.927991 392 external_mini_cluster.cc:1428] Setting key 6b514625fc3b9f067a1960536d945b24
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:44:37.940271 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.929339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.0.98.62:33697)
0929 14:44:37.929468 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.929471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.930147 (+ 676us) server_negotiation.cc:407] Connection header received
0929 14:44:37.931078 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.931082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.931200 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.931302 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.932172 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.932681 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.933756 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.934004 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.934094 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.936628 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.936632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.936634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.936661 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.938092 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.938670 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.938673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.938674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.938784 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.939137 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.939140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.939141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.939359 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.939453 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.939972 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.940100 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:44:37.941175 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.929447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.0.1:53964)
0929 14:44:37.929975 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.930033 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.930844 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.931384 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.931390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.931761 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.932023 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.932037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.932786 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.932789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.933638 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.933647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.934161 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.934165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.934308 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.934855 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.934870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.936462 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.938206 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.938213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.938224 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.938519 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.938897 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.938901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.938903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.939046 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.939464 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.939472 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.939728 (+ 256us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.939881 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.940223 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260929 14:44:37.941174 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.929447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.98.60:44145)
0929 14:44:37.930033 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.930047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.930084 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:44:37.930831 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.930856 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.931114 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.931264 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.932093 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.932977 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.933810 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.934187 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.934313 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.936982 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.936988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.936998 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.937020 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.938745 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.939136 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.939141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.939142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.939212 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.939448 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.939452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.939453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.939718 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.939899 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.940114 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.940310 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":475,"spinlock_wait_cycles":37504,"thread_start_us":148,"threads_started":1}
I20260929 14:44:37.942464 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.943326 13789 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 3
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:37.951968 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.942944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41291 (local address 127.0.0.1:56112)
0929 14:44:37.943215 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.943227 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.943319 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.943576 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.943579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.944004 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.944122 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.944128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.945100 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.945104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.945680 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.945688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.946307 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.946427 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.946961 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.946973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.948267 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.950046 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.950050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.950052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.950293 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.950620 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.950623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.950625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.950708 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.951409 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.951412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.951510 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.951608 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.951747 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2}
I20260929 14:44:37.951990 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.943078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.0.98.61:41291)
0929 14:44:37.943205 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.943211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.943267 (+ 56us) server_negotiation.cc:407] Connection header received
0929 14:44:37.943418 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.943422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.943475 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.943550 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.944225 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.944968 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.945791 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.946084 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946188 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.948402 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.948405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.948407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.948432 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.949947 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.950413 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.950417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.950419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.950513 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.951019 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.951021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.951023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.951265 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.951363 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.951679 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.951786 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:44:37.952783 13939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:44:37.952911 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.943662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33538)
0929 14:44:37.943772 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.943785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.943846 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.944129 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.944131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.944289 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.944368 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.944373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.945310 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.945313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.946048 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.946056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.946608 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946923 (+ 315us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.947055 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.947532 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.947543 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.949033 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.951053 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.951056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.951058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.951319 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.951679 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.951682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.951684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.951818 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.952357 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.952361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.952476 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.952552 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.952675 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":37}
I20260929 14:44:37.953912 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.943854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.0.98.60:44145)
0929 14:44:37.943939 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.943942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.943952 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:44:37.943979 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.943981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.944013 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.944092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.944472 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.945215 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946160 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.946497 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.949389 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.949393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.949395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.949416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.950849 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.951467 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.951470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.951472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.951569 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.951935 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.951938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.951940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.952248 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.953201 (+ 953us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.953271 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.953404 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":33,"spinlock_wait_cycles":1792}
I20260929 14:44:37.954198 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.943830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.98.62:33697)
0929 14:44:37.943920 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.943924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.943936 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:44:37.943970 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.943973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.944013 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.944092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.945119 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.945815 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.946672 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.947018 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.947100 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.950533 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.950536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.950539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.950565 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.952359 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.952831 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.952834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.952836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.953078 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.953379 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.953382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.953384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.953597 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.953842 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.953928 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.954054 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":32}
I20260929 14:44:37.955498 13864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } attempt: 3
I20260929 14:44:37.957389 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: Bootstrap starting.
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:44:37.960232 13789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: Neither blocks nor log segments found. Creating new log.
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:44:37.961210 13789 log.cc:824] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:37.961941 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.953287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33542)
0929 14:44:37.953446 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.953461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.953531 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.954345 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.954347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.954513 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.954603 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.954609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.955384 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.955387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.956090 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.956100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.956731 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.956734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.956804 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.957399 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.957414 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.958825 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.960451 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.960453 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.960456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.960627 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.960890 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.960893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.960895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.960970 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.961434 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.961437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.961522 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.961614 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.961756 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 14:44:37.961949 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.953329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.0.98.60:44145)
0929 14:44:37.954086 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.954090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.954102 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:44:37.954133 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.954136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.954180 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.954252 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.954710 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.955283 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.956210 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.956521 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.956630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.958937 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.958940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.958943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.958965 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.960350 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.960709 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.960712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.960714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.960783 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.961056 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.961058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.961060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.961326 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.961452 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.961663 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.961808 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":694,"spinlock_wait_cycles":2432,"thread_start_us":126,"threads_started":1}
I20260929 14:44:37.963850 13789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: No bootstrap required, opened a new log
I20260929 14:44:37.965458 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.955873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44145 (local address 127.0.0.1:33546)
0929 14:44:37.956040 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:37.956052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:37.956122 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:37.956604 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:37.956607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:37.956762 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:37.956850 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.956855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.957690 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.957693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.958410 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:37.958421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.959082 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.959086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:37.959173 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.959550 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:37.959564 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:37.961133 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:37.963831 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.963834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.963836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.964080 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.964434 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:37.964437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:37.964439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:37.964548 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:37.964991 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:37.964994 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:37.965086 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 14:44:37.965163 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 14:44:37.965292 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 14:44:37.965649 13939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: Bootstrap starting.
I20260929 14:44:37.966346 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.956285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.0.98.60:44145)
0929 14:44:37.956403 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.956412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.956422 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:44:37.956447 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.956449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.956486 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.956556 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.956984 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.957587 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.958546 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.958964 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.959063 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.961270 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.961273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.961276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.961377 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.963717 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.964252 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.964255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.964257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.964339 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.964658 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.964661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.964663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.964854 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.966007 (+ 1153us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.966076 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.966190 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 14:44:37 dist-test-slave-g8hp krb5kdc[13708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:44:37.968112 13789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.968345 13789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:37.968389 13789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138539603fe242eb90cfa14705930a10, State: Initialized, Role: FOLLOWER
I20260929 14:44:37.968722 13939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:37.968894 13789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.969852 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: Bootstrap starting.
I20260929 14:44:37.970112 13939 log.cc:824] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:37.970358 13789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:37.972551 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.961722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.0.98.61:41291)
0929 14:44:37.961846 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.961850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.961862 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:44:37.961900 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.961903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.961947 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.962050 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.963423 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.964149 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.965192 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.965480 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.965551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.968683 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.968686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.968689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.968712 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.970260 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.970799 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.970802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.970804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.970885 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:37.971196 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:37.971199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:37.971201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:37.971407 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:37.971566 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:37.972291 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 14:44:37.972404 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:44:37.972580 13864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:37.972995 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:37.973148 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:37.973511 13864 log.cc:824] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:37.975195 13939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: No bootstrap required, opened a new log
I20260929 14:44:37.975451 13864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: No bootstrap required, opened a new log
I20260929 14:44:37.976486 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:44:37.976622 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
W20260929 14:44:37.976992 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:44:37.977067 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:44:37.977833 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b28ca853dfa4404fb31c1c073f62d8ba (127.0.98.61:41291), 4b713b6d5127427a9dac8d8d43e636cf (127.0.98.60:44145)
I20260929 14:44:37.979254 13864 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.979259 13939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.979470 13939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:37.979470 13864 raft_consensus.cc:374] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:37.979519 13864 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28ca853dfa4404fb31c1c073f62d8ba, State: Initialized, Role: FOLLOWER
I20260929 14:44:37.979519 13939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b713b6d5127427a9dac8d8d43e636cf, State: Initialized, Role: FOLLOWER
I20260929 14:44:37.979987 13939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.979987 13864 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.977138 13721 master_runner.cc:186] Error getting master registration for 127.0.98.61:41291: OK, instance_id { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" instance_seqno: 1790693077705728 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:44:37.980824 13721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:44:37.981112 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:37.981241 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:37.981490 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:37.981623 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:37.982026 13864 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:37.982084 13838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138539603fe242eb90cfa14705930a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" is_pre_election: true
I20260929 14:44:37.982201 13939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:37.982373 13838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138539603fe242eb90cfa14705930a10 in term 0.
I20260929 14:44:37.982635 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138539603fe242eb90cfa14705930a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b713b6d5127427a9dac8d8d43e636cf" is_pre_election: true
I20260929 14:44:37.982800 13738 leader_election.cc:302] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138539603fe242eb90cfa14705930a10, b28ca853dfa4404fb31c1c073f62d8ba; no voters:
I20260929 14:44:37.982914 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 138539603fe242eb90cfa14705930a10 in term 0.
I20260929 14:44:37.983105 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:44:37.983191 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:44:37.983247 13946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:37.984659 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
I20260929 14:44:37.986033 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:44:37.984907 13984 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 14:44:37.986544 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [CANDIDATE]: Term 1 election: Requested vote from peers b28ca853dfa4404fb31c1c073f62d8ba (127.0.98.61:41291), 4b713b6d5127427a9dac8d8d43e636cf (127.0.98.60:44145)
I20260929 14:44:37.986801 13838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138539603fe242eb90cfa14705930a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b28ca853dfa4404fb31c1c073f62d8ba"
I20260929 14:44:37.986853 13913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "138539603fe242eb90cfa14705930a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b713b6d5127427a9dac8d8d43e636cf"
I20260929 14:44:37.986923 13838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:37.986966 13913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:37.988302 13913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138539603fe242eb90cfa14705930a10 in term 1.
I20260929 14:44:37.988303 13838 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 138539603fe242eb90cfa14705930a10 in term 1.
I20260929 14:44:37.988662 13738 leader_election.cc:302] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138539603fe242eb90cfa14705930a10, b28ca853dfa4404fb31c1c073f62d8ba; no voters:
I20260929 14:44:37.989007 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:37.989272 13946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [term 1 LEADER]: Becoming Leader. State: Replica: 138539603fe242eb90cfa14705930a10, State: Running, Role: LEADER
I20260929 14:44:37.989612 13946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } }
W20260929 14:44:37.990310 13978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:44:37.990392 13978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:44:37.991554 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138539603fe242eb90cfa14705930a10. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:37.991694 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:37.996305 13796 master_runner.cc:263] No leader master found from master b28ca853dfa4404fb31c1c073f62d8ba
I20260929 14:44:37.996385 13796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:44:37.998935 13993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:37.999965 13993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:38.001747 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.990389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.98.60:44145)
0929 14:44:37.990763 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.990768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.991314 (+ 546us) server_negotiation.cc:407] Connection header received
0929 14:44:37.991505 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.991509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.991561 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.991613 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:37.992656 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.993509 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.994707 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:37.995074 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:37.995152 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:37.997895 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:37.997899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:37.997902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:37.997930 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:37.999719 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.000226 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.000230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.000232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.000327 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.000661 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.000665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.000666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.000933 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:38.001072 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:38.001450 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 14:44:38.001586 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:44:38.005249 13913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [term 1 FOLLOWER]: Refusing update from remote peer 138539603fe242eb90cfa14705930a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:38.005270 13838 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [term 1 FOLLOWER]: Refusing update from remote peer 138539603fe242eb90cfa14705930a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:38.005913 13990 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:38.006150 13946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:38.010157 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138539603fe242eb90cfa14705930a10. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.010334 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:38.010290 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138539603fe242eb90cfa14705930a10. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.010416 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:38.012581 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.012743 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:38.013428 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.013563 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:38.014768 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.014896 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:38.015667 13993 catalog_manager.cc:1347] Generated new cluster ID: 8bc1f14fa6b2454d9dcbc6b3b9b2eeab
I20260929 14:44:38.015726 13993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:38.016310 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "138539603fe242eb90cfa14705930a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "138539603fe242eb90cfa14705930a10" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } peers { permanent_uuid: "b28ca853dfa4404fb31c1c073f62d8ba" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41291 } } peers { permanent_uuid: "4b713b6d5127427a9dac8d8d43e636cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44145 } } }
I20260929 14:44:38.016423 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:44:38.022290 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.996040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56116 (local address 127.0.98.61:41291)
0929 14:44:37.996224 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.996227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:38.002359 (+ 6132us) server_negotiation.cc:407] Connection header received
0929 14:44:38.002559 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:38.002562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:38.002619 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:38.002670 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:38.003451 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:38.004247 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:38.013666 (+ 9419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:38.014021 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:38.014117 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:38.018422 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:38.018427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:38.018430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:38.018460 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:38.020425 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.020976 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.020980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.020982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.021075 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.021348 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.021350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.021352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.021560 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:38.021673 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:38.022007 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 14:44:38.022138 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 14:44:38.023958 13998 mvcc.cc:205] Tried to move back new op lower bound from 7334678847502761984 to 7334678847448776704. Current Snapshot: MvccSnapshot[applied={T|T < 7334678847502761984}]
I20260929 14:44:38.025815 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:37.993242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.0.98.62:33697)
0929 14:44:37.993365 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:37.993369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:37.996581 (+ 3212us) server_negotiation.cc:407] Connection header received
0929 14:44:37.998376 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:37.998380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:37.998437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:37.998488 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:38.011550 (+ 13062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:38.012339 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:38.016246 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:38.016587 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:38.016657 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:38.018872 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:38.018877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:38.018880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:38.018908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:38.021201 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.024277 (+ 3076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.024282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.024284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.024399 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:38.024730 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:38.024734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:38.024736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:38.025000 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:38.025148 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:38.025502 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 14:44:38.025642 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:44:38.047626 13993 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:44:38.049094 13993 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:38.060366 13993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138539603fe242eb90cfa14705930a10: Generated new TSK 0
I20260929 14:44:38.061395 13993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:44:38.989389 13984 catalog_manager.cc:1259] Loaded cluster ID: 8bc1f14fa6b2454d9dcbc6b3b9b2eeab
I20260929 14:44:38.989472 13984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: loading cluster ID for follower catalog manager: success
I20260929 14:44:38.991180 13984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: acquiring CA information for follower catalog manager: success
I20260929 14:44:38.991727 13978 catalog_manager.cc:1259] Loaded cluster ID: 8bc1f14fa6b2454d9dcbc6b3b9b2eeab
I20260929 14:44:38.991801 13978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: loading cluster ID for follower catalog manager: success
I20260929 14:44:38.992443 13984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b713b6d5127427a9dac8d8d43e636cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:44:38.994283 13978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: acquiring CA information for follower catalog manager: success
I20260929 14:44:38.995814 13978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b28ca853dfa4404fb31c1c073f62d8ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:44:39.126879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13721
I20260929 14:44:39.132561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13796
I20260929 14:44:39.137842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 13870
2026-09-29T14:44:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2954 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:44:39 dist-test-slave-g8hp krb5kdc[14011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:44:39 dist-test-slave-g8hp krb5kdc[14011](info): set up 2 sockets
Sep 29 14:44:39 dist-test-slave-g8hp krb5kdc[14011](info): commencing operation
krb5kdc: starting...
W20260929 14:44:40.101052 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:44:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:44:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:44:40.220886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:40.331530 14024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:40.331828 14024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:40.331887 14024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:40.336010 14024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:40.336073 14024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:40.336098 14024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:40.336138 14024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:40.336167 14024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:40.341624 14024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:40.342700 14024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:40.343707 14024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:40.352468 14029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:40.352454 14032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:40.352653 14024 server_base.cc:1034] running on GCE node
W20260929 14:44:40.352454 14030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:40.353444 14024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:40.354604 14024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:40.355779 14024 hybrid_clock.cc:648] HybridClock initialized: now 1790693080355745 us; error 47 us; skew 500 ppm
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:40.358628 14024 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:40.359560 14024 webserver.cc:460] Webserver started at http://127.0.98.62:33499/ using document root <none> and password file <none>
I20260929 14:44:40.360091 14024 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:40.360162 14024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:40.360327 14024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:40.362478 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "3dbba2443bf742d7801302de8c7cd14a"
format_stamp: "Formatted at 2026-09-29 14:44:40 on dist-test-slave-g8hp"
server_key: "aa775ba876779a6278540ff56cbafa0b"
server_key_iv: "d9e782cb5fcc7b45e4a6f97770e05499"
server_key_version: "encryptionkey@0"
I20260929 14:44:40.363178 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "3dbba2443bf742d7801302de8c7cd14a"
format_stamp: "Formatted at 2026-09-29 14:44:40 on dist-test-slave-g8hp"
server_key: "aa775ba876779a6278540ff56cbafa0b"
server_key_iv: "d9e782cb5fcc7b45e4a6f97770e05499"
server_key_version: "encryptionkey@0"
I20260929 14:44:40.367810 14024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:44:40.371124 14039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:40.372653 14024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:40.372809 14024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "3dbba2443bf742d7801302de8c7cd14a"
format_stamp: "Formatted at 2026-09-29 14:44:40 on dist-test-slave-g8hp"
server_key: "aa775ba876779a6278540ff56cbafa0b"
server_key_iv: "d9e782cb5fcc7b45e4a6f97770e05499"
server_key_version: "encryptionkey@0"
I20260929 14:44:40.372982 14024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:40.388311 14024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:40.391762 14024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:40.391961 14024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:40.399663 14024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36083
I20260929 14:44:40.399696 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36083 every 8 connection(s)
I20260929 14:44:40.400686 14024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:40.403728 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:40.407366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14024
I20260929 14:44:40.407510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:44:40.407812 392 external_mini_cluster.cc:1428] Setting key 805d71825c5db048527e25df4690d021
I20260929 14:44:40.410149 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a: Bootstrap starting.
I20260929 14:44:40.412529 14092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:40.413362 14092 log.cc:824] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:40.415678 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a: No bootstrap required, opened a new log
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:40.419844 14092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260929 14:44:40.420117 14092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:40.420189 14092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbba2443bf742d7801302de8c7cd14a, State: Initialized, Role: FOLLOWER
I20260929 14:44:40.420723 14092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260929 14:44:40.420903 14092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:40.421202 14092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:40.421316 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:40.422294 14092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260929 14:44:40.422641 14092 leader_election.cc:302] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dbba2443bf742d7801302de8c7cd14a; no voters:
I20260929 14:44:40.422959 14092 leader_election.cc:288] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:44:40.423074 14097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:40.423321 14097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [term 1 LEADER]: Becoming Leader. State: Replica: 3dbba2443bf742d7801302de8c7cd14a, State: Running, Role: LEADER
I20260929 14:44:40.423714 14097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } }
I20260929 14:44:40.423887 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:40.409461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.0.98.62:36083)
0929 14:44:40.409988 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:40.410003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:40.410060 (+ 57us) server_negotiation.cc:407] Connection header received
0929 14:44:40.410793 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:40.410814 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:40.411187 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:40.411537 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:40.412393 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:40.413534 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:40.414552 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:40.415063 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:40.415225 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:40.418227 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:40.418233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:40.418249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:40.418283 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:40.420019 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:40.420655 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:40.420661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:40.420666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:40.420935 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:40.421340 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:40.421343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:40.421344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:40.421838 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:40.422035 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:40.422354 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 14:44:40.422798 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":155,"threads_started":1}
I20260929 14:44:40.424355 14092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:40.425287 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dbba2443bf742d7801302de8c7cd14a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260929 14:44:40.425426 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:40.425804 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dbba2443bf742d7801302de8c7cd14a. Latest consensus state: current_term: 1 leader_uuid: "3dbba2443bf742d7801302de8c7cd14a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dbba2443bf742d7801302de8c7cd14a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36083 } } }
I20260929 14:44:40.425848 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:40.425931 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:40.427342 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:44:40.430251 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:40.436264 14101 catalog_manager.cc:1347] Generated new cluster ID: 830c04e5860a420e9e2b04ae03654778
I20260929 14:44:40.436345 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:40.446236 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:44:40.447680 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:40.454771 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dbba2443bf742d7801302de8c7cd14a: Generated new TSK 0
I20260929 14:44:40.455662 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:44:40.483613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14024
2026-09-29T14:44:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1344 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14119](info): set up 2 sockets
Sep 29 14:44:40 dist-test-slave-g8hp krb5kdc[14119](info): commencing operation
krb5kdc: starting...
W20260929 14:44:41.436957 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:44:41 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:44:41.560741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:41.670063 14132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:41.670387 14132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:41.670428 14132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:41.674510 14132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:41.674582 14132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:41.674599 14132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:41.674614 14132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:41.674628 14132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:41.680413 14132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:41.681687 14132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:41.682761 14132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:41.691658 14140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:41.691713 14132 server_base.cc:1034] running on GCE node
W20260929 14:44:41.691663 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:41.691658 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:41.692713 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:41.693989 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:41.695175 14132 hybrid_clock.cc:648] HybridClock initialized: now 1790693081695150 us; error 40 us; skew 500 ppm
Sep 29 14:44:41 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:41.704556 14132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:41.705592 14132 webserver.cc:460] Webserver started at http://127.0.98.62:33533/ using document root <none> and password file <none>
I20260929 14:44:41.706166 14132 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:41.706239 14132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:41.706457 14132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:41.708634 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "2660ab6d618a4f7da73a8bf6869eceb0"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "f33853993d35cedbcdb9cebc58e17f0a"
server_key_iv: "6366119caea38cedeea204bb78643f3d"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.709250 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2660ab6d618a4f7da73a8bf6869eceb0"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "f33853993d35cedbcdb9cebc58e17f0a"
server_key_iv: "6366119caea38cedeea204bb78643f3d"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.713275 14132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:44:41.715896 14147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:41.717381 14132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:44:41.717514 14132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "2660ab6d618a4f7da73a8bf6869eceb0"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "f33853993d35cedbcdb9cebc58e17f0a"
server_key_iv: "6366119caea38cedeea204bb78643f3d"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.717624 14132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:41.748472 14132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:41.752007 14132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:41.752208 14132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:41.759474 14132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40209
I20260929 14:44:41.759774 14199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40209 every 8 connection(s)
I20260929 14:44:41.760479 14132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:41.763528 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:41.767499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14132
I20260929 14:44:41.767632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:44:41.767885 392 external_mini_cluster.cc:1428] Setting key d91279b3171fe4f1e793e49672cb5520
I20260929 14:44:41.769599 14200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap starting.
I20260929 14:44:41.772423 14200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:41.773442 14200 log.cc:824] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:41.776021 14200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: No bootstrap required, opened a new log
Sep 29 14:44:41 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:41.780674 14200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:41.781059 14200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:41.781119 14200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Initialized, Role: FOLLOWER
I20260929 14:44:41.781621 14200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:41.781765 14200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:41.781836 14200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:41.781941 14200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:41.782971 14200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:41.783380 14200 leader_election.cc:302] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660ab6d618a4f7da73a8bf6869eceb0; no voters:
I20260929 14:44:41.783874 14200 leader_election.cc:288] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:44:41.783968 14205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:41.784138 14205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 LEADER]: Becoming Leader. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Running, Role: LEADER
I20260929 14:44:41.784162 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:41.769399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.62:40209)
0929 14:44:41.769852 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:41.769874 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:41.769917 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:44:41.770771 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:41.770798 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:41.771247 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:41.771616 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:41.772308 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:41.773907 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:41.774987 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:41.775665 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:41.775836 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:41.778483 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:41.778490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:41.778511 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:41.778546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:41.780655 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:41.781248 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:41.781254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:41.781260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:41.781408 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:41.781701 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:41.781704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:41.781706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:41.782267 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:41.782543 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:41.782801 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 14:44:41.783158 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":119,"threads_started":1}
I20260929 14:44:41.784552 14205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:41.785424 14200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:41.787585 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660ab6d618a4f7da73a8bf6869eceb0. Latest consensus state: current_term: 1 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:41.787737 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:41.788021 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:41.788167 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:41.788390 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:41.791430 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:41.798038 14220 catalog_manager.cc:1347] Generated new cluster ID: b8cd32d365614fd8b45304404b1c4b34
I20260929 14:44:41.798126 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:41.806054 14220 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:44:41.807152 14220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:41.813515 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Generated new TSK 0
I20260929 14:44:41.814266 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:44:41.844530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:41.956352 14226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:41.956630 14226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:41.956676 14226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:41.960804 14226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:41.960898 14226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:41.960996 14226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:44:41.966504 14226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:41.967643 14226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:41.968818 14226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:41.979339 14231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:41.979401 14232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:41.979339 14234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:41.979408 14226 server_base.cc:1034] running on GCE node
I20260929 14:44:41.979997 14226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:41.980634 14226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:41.981792 14226 hybrid_clock.cc:648] HybridClock initialized: now 1790693081981751 us; error 54 us; skew 500 ppm
Sep 29 14:44:41 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:41.984539 14226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:44:41.985502 14226 webserver.cc:460] Webserver started at http://127.0.98.1:42069/ using document root <none> and password file <none>
I20260929 14:44:41.986039 14226 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:41.986107 14226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:41.986312 14226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:41.988386 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "9c2416daec5faee79374a05e5d67571a"
server_key_iv: "7dffa386034330c164679683dd5e5c19"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.988955 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "9c2416daec5faee79374a05e5d67571a"
server_key_iv: "7dffa386034330c164679683dd5e5c19"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.992748 14226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 14:44:41.995193 14241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:41.996459 14226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:44:41.996572 14226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "9c2416daec5faee79374a05e5d67571a"
server_key_iv: "7dffa386034330c164679683dd5e5c19"
server_key_version: "encryptionkey@0"
I20260929 14:44:41.996671 14226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:42.014585 14226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:42.017803 14226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:42.017984 14226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:42.018573 14226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:42.019403 14226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:42.019457 14226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.019511 14226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:42.019536 14226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.028416 14226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44693
I20260929 14:44:42.028439 14354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44693 every 8 connection(s)
I20260929 14:44:42.029323 14226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:44:42.030697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14226
I20260929 14:44:42.030834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:44:42.031221 392 external_mini_cluster.cc:1428] Setting key b60e3cf0c67584cdb95e8a74774d7d30
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:44:42.043704 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.031276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43925 (local address 127.0.98.62:40209)
0929 14:44:42.031481 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.031484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.032614 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 14:44:42.033631 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.033635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.033689 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.033759 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:42.034915 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.035680 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.036937 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.037296 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.037413 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.039885 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:42.039889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:42.039891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:42.039917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:42.041348 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.041908 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.041911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.041912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.041998 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.042335 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.042339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.042342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.042571 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:42.042682 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.043442 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.043565 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260929 14:44:42.044711 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.031788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:43925)
0929 14:44:42.032470 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.032521 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.033422 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.033943 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.033950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.034293 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:42.034773 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.034786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.035801 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.035804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.036773 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.036783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.037440 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.037443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.037588 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.038149 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:42.038166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:42.039691 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:42.041459 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.041465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.041479 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.041806 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.042076 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.042079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.042080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.042223 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.042727 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:42.042733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:42.043077 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.043305 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.043688 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":114,"threads_started":1}
I20260929 14:44:42.045990 14355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
I20260929 14:44:42.046233 14355 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:42.046742 14355 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:42.048229 14164 ts_manager.cc:194] Registered new tserver with Master: cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
I20260929 14:44:42.049424 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:44:42.059657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:42.172343 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:42.172621 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:42.172667 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:42.176920 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:42.176985 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:42.177047 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:44:42.182328 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:42.183385 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:42.184480 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:42.194725 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.194725 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.194725 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.194972 14360 server_base.cc:1034] running on GCE node
I20260929 14:44:42.195466 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:42.196192 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:42.197350 14360 hybrid_clock.cc:648] HybridClock initialized: now 1790693082197318 us; error 43 us; skew 500 ppm
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:42.200296 14360 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:44:42.201308 14360 webserver.cc:460] Webserver started at http://127.0.98.2:46471/ using document root <none> and password file <none>
I20260929 14:44:42.201891 14360 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:42.201962 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:42.202173 14360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:42.204305 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "9ad9361c9ae44b66b60d40cf53e38629"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "29e95d977dbc1cc21d0473244023e3a3"
server_key_iv: "e9d53c2b025dc73b28ea939183cd3deb"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.204855 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9ad9361c9ae44b66b60d40cf53e38629"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "29e95d977dbc1cc21d0473244023e3a3"
server_key_iv: "e9d53c2b025dc73b28ea939183cd3deb"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.208602 14360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:44:42.211254 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.212610 14360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 14:44:42.212738 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "9ad9361c9ae44b66b60d40cf53e38629"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "29e95d977dbc1cc21d0473244023e3a3"
server_key_iv: "e9d53c2b025dc73b28ea939183cd3deb"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.212882 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:42.235888 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:42.239686 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:42.239885 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:42.240492 14360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:42.241380 14360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:42.241428 14360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.241480 14360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:42.241509 14360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.250244 14360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44899
I20260929 14:44:42.250264 14488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44899 every 8 connection(s)
I20260929 14:44:42.251171 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:44:42.256500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14360
I20260929 14:44:42.256630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:44:42.256960 392 external_mini_cluster.cc:1428] Setting key 03c377bd579636e8372e590e6a09c989
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 14:44:42.266935 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.253351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52559 (local address 127.0.98.62:40209)
0929 14:44:42.253482 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.253485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.254397 (+ 912us) server_negotiation.cc:407] Connection header received
0929 14:44:42.255653 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.255656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.255713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.255788 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:42.257369 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.258047 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.259736 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.259977 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.260064 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.262948 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:42.262953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:42.262956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:42.262988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:42.264648 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.265244 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.265248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.265250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.265343 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.265694 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.265697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.265699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.265966 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:42.266072 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.266608 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.266764 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:44:42.267921 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.253768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:52559)
0929 14:44:42.254237 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.254281 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.255441 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.255980 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.255987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.256447 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:42.257165 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.257188 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.258183 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.258187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.259595 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.259607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.260137 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.260140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.260366 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.261138 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:42.261156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:42.262699 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:42.264802 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.264808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.264823 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.265143 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.265440 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.265443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.265444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.265589 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.266075 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:42.266080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:42.266343 (+ 263us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.266532 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.266904 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260929 14:44:42.269560 14489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:44:42.269887 14489 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:42.270469 14489 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:42.271656 14164 ts_manager.cc:194] Registered new tserver with Master: 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899)
I20260929 14:44:42.272235 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:44:42.288573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:42.399318 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:42.399633 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:42.399704 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:42.404093 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:42.404163 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:42.404244 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:44:42.409672 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:42.410809 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:42.411899 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:42.421767 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.421772 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.421770 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.422199 14494 server_base.cc:1034] running on GCE node
I20260929 14:44:42.422591 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:42.423265 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:42.424437 14494 hybrid_clock.cc:648] HybridClock initialized: now 1790693082424423 us; error 27 us; skew 500 ppm
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:42.427156 14494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:44:42.428042 14494 webserver.cc:460] Webserver started at http://127.0.98.3:46365/ using document root <none> and password file <none>
I20260929 14:44:42.428588 14494 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:42.428655 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:42.428848 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:44:42.430935 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "61bbd520b21f4a24b5390a6db1e0a98c"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "882830a1c42012f6fb62c99dd3f503e1"
server_key_iv: "93ab3adc0fcfdd693107f3b99a0216cc"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.431490 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "61bbd520b21f4a24b5390a6db1e0a98c"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "882830a1c42012f6fb62c99dd3f503e1"
server_key_iv: "93ab3adc0fcfdd693107f3b99a0216cc"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.435482 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 14:44:42.438040 14509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.439272 14494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 14:44:42.439378 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "61bbd520b21f4a24b5390a6db1e0a98c"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "882830a1c42012f6fb62c99dd3f503e1"
server_key_iv: "93ab3adc0fcfdd693107f3b99a0216cc"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.439476 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:42.456218 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:42.459362 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:42.459544 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:42.460139 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:42.460984 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:42.461030 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.461081 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:42.461110 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.469491 14494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33887
I20260929 14:44:42.469532 14622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33887 every 8 connection(s)
I20260929 14:44:42.470492 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:44:42.474906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14494
I20260929 14:44:42.475056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:44:42.475343 392 external_mini_cluster.cc:1428] Setting key a2021a8bee0a38dcd148e3b7f9df29cb
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:42.484637 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.472415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33433 (local address 127.0.98.62:40209)
0929 14:44:42.472587 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.472591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.473737 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 14:44:42.474627 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.474630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.474680 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.474755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:42.476262 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.477004 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.478414 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.478632 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.478719 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.481364 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:42.481367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:42.481370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:42.481397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:42.482770 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.483259 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.483261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.483263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.483355 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.483636 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.483638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.483639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.483810 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:42.483884 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.484396 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.484511 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 14:44:42.485577 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.472802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:33433)
0929 14:44:42.473579 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.473623 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.474419 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.474955 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.474965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.475471 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:42.476106 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.476123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.477140 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.477143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.478298 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.478309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.478757 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.478760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.478951 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.479521 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:42.479538 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:42.481123 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:42.482884 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.482889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.482902 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.483167 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.483424 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:42.483426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:42.483428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:42.483571 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:42.483891 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:42.483896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:42.484150 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.484320 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.484589 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":110,"threads_started":1}
I20260929 14:44:42.486884 14623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
I20260929 14:44:42.487106 14623 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:42.487594 14623 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:42.488636 14164 ts_manager.cc:194] Registered new tserver with Master: 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887)
I20260929 14:44:42.489413 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:44:42.489400 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33433
I20260929 14:44:42.498661 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.491341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.0.98.62:40209)
0929 14:44:42.491453 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.491457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.491670 (+ 213us) server_negotiation.cc:407] Connection header received
0929 14:44:42.491782 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.491785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.491828 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.491898 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:42.492584 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.493136 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.494006 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.494217 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.494304 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.495637 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:42.495641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:42.495643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:42.495667 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:42.497025 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.497435 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.497439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.497440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.497517 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:42.497758 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:42.497762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:42.497763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:42.497993 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:42.498110 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.498381 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.498501 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:44:42.501972 14164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:44:42.505225 14164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:44:42.520366 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.513897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.98.1:44693)
0929 14:44:42.514492 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.514509 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.514823 (+ 314us) server_negotiation.cc:407] Connection header received
0929 14:44:42.514912 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.514919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.515141 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.515293 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:42.515791 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517039 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519137 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.519688 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519795 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.519953 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.520099 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.520205 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":77,"threads_started":1}
I20260929 14:44:42.520356 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.513824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33887 (local address 127.0.0.1:35158)
0929 14:44:42.515057 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.515070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.515144 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.515635 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.515638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.515653 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:42.515768 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.515774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517263 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.517266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.518829 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.518846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.519907 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.520003 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.520025 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.520074 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.520217 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":524,"thread_start_us":78,"threads_started":1}
I20260929 14:44:42.520648 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.514773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.98.2:44899)
0929 14:44:42.515699 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.515711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.515733 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:44:42.515802 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.515807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.515968 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.516090 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:42.516319 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517099 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519066 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.519786 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519885 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.520022 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.520419 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.520512 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":147,"threads_started":1}
I20260929 14:44:42.520835 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.513783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:52874)
0929 14:44:42.514492 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.514546 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.514680 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.515495 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.515501 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.515525 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:42.515662 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.515669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517204 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.517207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.519017 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.519026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.519805 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.519808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.519913 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.519948 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.520057 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.520703 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":308,"spinlock_wait_cycles":10496,"thread_start_us":127,"threads_started":1}
I20260929 14:44:42.521014 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.514555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.0.1:36474)
0929 14:44:42.515415 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.515425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.515487 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.516122 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.516126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.516144 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:42.516216 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.516221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517225 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.517228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.518943 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.518954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.520076 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.520080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.520167 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.520187 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.520243 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.520312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":304,"thread_start_us":95,"threads_started":1}
I20260929 14:44:42.521528 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.514712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.0.98.3:33887)
0929 14:44:42.515215 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.515228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.515250 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:44:42.515320 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.515325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.515496 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.515642 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:42.515882 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.517085 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.518957 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.519770 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.520210 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.520337 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.520397 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.521393 (+ 996us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":91,"threads_started":1}
I20260929 14:44:42.523998 14423 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce (DEFAULT_TABLE table=test-table [id=9b8bca9f3e7349f0a2ac46f28bd3e9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:42.523998 14289 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce (DEFAULT_TABLE table=test-table [id=9b8bca9f3e7349f0a2ac46f28bd3e9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:42.525216 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5943adec224c4eb646b0b6c7d917ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:42.525216 14423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5943adec224c4eb646b0b6c7d917ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:42.526676 14557 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce (DEFAULT_TABLE table=test-table [id=9b8bca9f3e7349f0a2ac46f28bd3e9a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:42.527552 14557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5943adec224c4eb646b0b6c7d917ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:42.533385 14640 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap starting.
I20260929 14:44:42.533564 14641 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Bootstrap starting.
I20260929 14:44:42.534447 14642 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Bootstrap starting.
I20260929 14:44:42.535988 14640 tablet_bootstrap.cc:654] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:42.536113 14642 tablet_bootstrap.cc:654] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:42.536669 14641 tablet_bootstrap.cc:654] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:42.537180 14642 log.cc:824] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:42.537245 14640 log.cc:824] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:42.537405 14641 log.cc:824] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:42.539083 14640 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: No bootstrap required, opened a new log
I20260929 14:44:42.539093 14642 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: No bootstrap required, opened a new log
I20260929 14:44:42.539255 14642 ts_tablet_manager.cc:1397] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:44:42.539258 14640 ts_tablet_manager.cc:1397] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:44:42.542204 14640 raft_consensus.cc:348] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.542435 14640 raft_consensus.cc:374] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:42.542493 14640 raft_consensus.cc:729] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf608b45f327438ca6a3931b1a8cdc5f, State: Initialized, Role: FOLLOWER
I20260929 14:44:42.542563 14641 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: No bootstrap required, opened a new log
I20260929 14:44:42.542379 14642 raft_consensus.cc:348] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.542629 14642 raft_consensus.cc:374] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:42.542683 14642 raft_consensus.cc:729] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bbd520b21f4a24b5390a6db1e0a98c, State: Initialized, Role: FOLLOWER
I20260929 14:44:42.542807 14641 ts_tablet_manager.cc:1397] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 14:44:42.542974 14640 consensus_queue.cc:260] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.543154 14642 consensus_queue.cc:260] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.543752 14355 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
I20260929 14:44:42.543980 14640 ts_tablet_manager.cc:1428] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:44:42.544128 14642 ts_tablet_manager.cc:1428] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:44:42.544441 14623 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
I20260929 14:44:42.546970 14641 raft_consensus.cc:348] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.547190 14641 raft_consensus.cc:374] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:42.547245 14641 raft_consensus.cc:729] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad9361c9ae44b66b60d40cf53e38629, State: Initialized, Role: FOLLOWER
I20260929 14:44:42.547643 14641 consensus_queue.cc:260] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.548230 14489 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
I20260929 14:44:42.548522 14641 ts_tablet_manager.cc:1428] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260929 14:44:42.580689 14647 raft_consensus.cc:482] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:44:42.580951 14647 raft_consensus.cc:504] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.582235 14647 leader_election.cc:288] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693), 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899)
I20260929 14:44:42.586247 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.582789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34865 (local address 127.0.98.1:44693)
0929 14:44:42.582902 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.582906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.582918 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:44:42.582967 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.582970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.583010 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.583106 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:42.583381 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.584025 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585231 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.585703 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.585866 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.586037 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.586115 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:44:42.586247 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.582523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.98.3:34865)
0929 14:44:42.582698 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.582714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.582818 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.583113 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.583116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.583143 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:42.583271 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.583278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.584132 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.584135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.585105 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.585116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.585817 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.585927 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.585949 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.586013 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.586089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":3}
I20260929 14:44:42.586272 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.582635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59459 (local address 127.0.98.2:44899)
0929 14:44:42.582807 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.582812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.583047 (+ 235us) server_negotiation.cc:407] Connection header received
0929 14:44:42.583127 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.583129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.583177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.583250 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:42.583489 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.584113 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585294 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.585737 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585840 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.585899 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.586064 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.586138 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 14:44:42.586310 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.582523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.98.3:59459)
0929 14:44:42.582968 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.582978 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:42.583053 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:42.583267 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:42.583270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:42.583283 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:42.583379 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.583385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.584257 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.584260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.585182 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:42.585194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.585854 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.585858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:42.585955 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.585978 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:44:42.586042 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:44:42.586111 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":34,"thread_start_us":102,"threads_started":1}
I20260929 14:44:42.586947 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5943adec224c4eb646b0b6c7d917ce" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" is_pre_election: true
I20260929 14:44:42.587064 14443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5943adec224c4eb646b0b6c7d917ce" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad9361c9ae44b66b60d40cf53e38629" is_pre_election: true
I20260929 14:44:42.587321 14443 raft_consensus.cc:2454] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 0.
I20260929 14:44:42.587321 14309 raft_consensus.cc:2454] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 0.
I20260929 14:44:42.587782 14511 leader_election.cc:302] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bbd520b21f4a24b5390a6db1e0a98c, 9ad9361c9ae44b66b60d40cf53e38629; no voters:
I20260929 14:44:42.588239 14647 raft_consensus.cc:2781] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:44:42.588310 14647 raft_consensus.cc:482] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:44:42.588338 14647 raft_consensus.cc:3037] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:42.589442 14647 raft_consensus.cc:504] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.589793 14647 leader_election.cc:288] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 election: Requested vote from peers cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693), 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899)
I20260929 14:44:42.590165 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5943adec224c4eb646b0b6c7d917ce" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
I20260929 14:44:42.590165 14443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5943adec224c4eb646b0b6c7d917ce" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad9361c9ae44b66b60d40cf53e38629"
I20260929 14:44:42.590313 14443 raft_consensus.cc:3037] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:42.590313 14309 raft_consensus.cc:3037] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:42.591596 14443 raft_consensus.cc:2454] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 1.
I20260929 14:44:42.591596 14309 raft_consensus.cc:2454] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 1.
I20260929 14:44:42.591899 14511 leader_election.cc:302] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bbd520b21f4a24b5390a6db1e0a98c, 9ad9361c9ae44b66b60d40cf53e38629; no voters:
I20260929 14:44:42.592079 14647 raft_consensus.cc:2781] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:42.592240 14647 raft_consensus.cc:686] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 LEADER]: Becoming Leader. State: Replica: 61bbd520b21f4a24b5390a6db1e0a98c, State: Running, Role: LEADER
I20260929 14:44:42.592604 14647 consensus_queue.cc:237] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:42.595656 14163 catalog_manager.cc:5495] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } health_report { overall_health: UNKNOWN } } }
I20260929 14:44:42.607676 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.604330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.98.3:33887)
0929 14:44:42.604438 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:42.604442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:42.604547 (+ 105us) server_negotiation.cc:407] Connection header received
0929 14:44:42.604676 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:42.604678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:42.604730 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:42.604816 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:44:42.605018 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.605547 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.606381 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:42.606578 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:42.606644 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:42.606909 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:44:42.607317 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:44:42.607440 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:42.607502 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 14:44:42.607553 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:44:42.613546 14443 raft_consensus.cc:1261] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Refusing update from remote peer 61bbd520b21f4a24b5390a6db1e0a98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:42.613545 14309 raft_consensus.cc:1261] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Refusing update from remote peer 61bbd520b21f4a24b5390a6db1e0a98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:42.614203 14650 consensus_queue.cc:1035] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:42.614464 14647 consensus_queue.cc:1035] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:42.623077 14654 mvcc.cc:205] Tried to move back new op lower bound from 7334678866378342400 to 7334678866301968384. Current Snapshot: MvccSnapshot[applied={T|T < 7334678866378342400}]
I20260929 14:44:42.630956 14164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:42.631120 14164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:42.633531 14164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Sending DeleteTablet for 3 replicas of tablet 9e5943adec224c4eb646b0b6c7d917ce
I20260929 14:44:42.634316 14289 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52874
I20260929 14:44:42.634418 14423 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36474
I20260929 14:44:42.634599 14557 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35158
I20260929 14:44:42.634868 14663 tablet_replica.cc:330] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: stopping tablet replica
I20260929 14:44:42.634912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14226
I20260929 14:44:42.635197 14663 raft_consensus.cc:2229] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:42.635521 14664 tablet_replica.cc:330] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: stopping tablet replica
I20260929 14:44:42.635811 14664 raft_consensus.cc:2229] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:42.636109 14664 raft_consensus.cc:2258] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:42.637657 14664 ts_tablet_manager.cc:1905] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:42.640520 14664 ts_tablet_manager.cc:1918] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:42.640702 14664 log.cc:1196] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/9e5943adec224c4eb646b0b6c7d917ce
I20260929 14:44:42.641122 14664 ts_tablet_manager.cc:1939] T 9e5943adec224c4eb646b0b6c7d917ce P 9ad9361c9ae44b66b60d40cf53e38629: Deleting consensus metadata
I20260929 14:44:42.641130 14665 tablet_replica.cc:330] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: stopping tablet replica
W20260929 14:44:42.641292 14151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44693 (error 108)
I20260929 14:44:42.641469 14665 raft_consensus.cc:2229] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:44:42.641913 14665 raft_consensus.cc:2258] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:42.642131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14360
I20260929 14:44:42.644629 14665 ts_tablet_manager.cc:1905] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:42.647459 14665 ts_tablet_manager.cc:1918] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:42.647660 14665 log.cc:1196] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/9e5943adec224c4eb646b0b6c7d917ce
I20260929 14:44:42.648016 14665 ts_tablet_manager.cc:1939] T 9e5943adec224c4eb646b0b6c7d917ce P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting consensus metadata
I20260929 14:44:42.649336 14149 catalog_manager.cc:4841] TS 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887): tablet 9e5943adec224c4eb646b0b6c7d917ce (table test-table [id=9b8bca9f3e7349f0a2ac46f28bd3e9a0]) successfully deleted
I20260929 14:44:42.649433 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.649162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:38754)
0929 14:44:42.649278 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.649355 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 14:44:42.649585 14151 catalog_manager.cc:4584] TS cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e5943adec224c4eb646b0b6c7d917ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
I20260929 14:44:42.650005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14494
I20260929 14:44:42.651595 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:42.650056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.0.1:59596)
0929 14:44:42.651475 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:42.651526 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1343}
I20260929 14:44:42.655184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14132
I20260929 14:44:42.660292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--webserver_interface=127.0.98.62
--webserver_port=33533
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:42.770071 14667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:42.770390 14667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:42.770448 14667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:42.774574 14667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:42.774652 14667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:42.774678 14667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:42.774699 14667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:42.774718 14667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:42.780117 14667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:42.781387 14667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:42.782500 14667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:42.795706 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.795758 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.795774 14667 server_base.cc:1034] running on GCE node
W20260929 14:44:42.795710 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.796665 14667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:42.797868 14667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:42.799036 14667 hybrid_clock.cc:648] HybridClock initialized: now 1790693082799016 us; error 33 us; skew 500 ppm
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:42.802057 14667 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:42.803045 14667 webserver.cc:460] Webserver started at http://127.0.98.62:33533/ using document root <none> and password file <none>
I20260929 14:44:42.803617 14667 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:42.803699 14667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:42.807587 14667 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:44:42.809762 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:42.810957 14667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:44:42.811072 14667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "2660ab6d618a4f7da73a8bf6869eceb0"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "f33853993d35cedbcdb9cebc58e17f0a"
server_key_iv: "6366119caea38cedeea204bb78643f3d"
server_key_version: "encryptionkey@0"
I20260929 14:44:42.811651 14667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:42.827070 14667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:42.830291 14667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:42.830490 14667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:42.837604 14667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40209
I20260929 14:44:42.837864 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40209 every 8 connection(s)
I20260929 14:44:42.838754 14667 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:42.843482 14736 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:44:42.845376 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap starting.
I20260929 14:44:42.847012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14667
I20260929 14:44:42.847491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44693
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=42069
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:42.863114 14736 log.cc:824] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:42.872476 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:42.872921 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap complete.
I20260929 14:44:42.876969 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:42.877413 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Initialized, Role: FOLLOWER
I20260929 14:44:42.877923 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:42.878042 14736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:42.878111 14736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:42.878206 14736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:44:42.880196 14736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:42.880527 14736 leader_election.cc:302] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660ab6d618a4f7da73a8bf6869eceb0; no voters:
I20260929 14:44:42.880820 14736 leader_election.cc:288] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:44:42.880986 14740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:44:42.881239 14740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 LEADER]: Becoming Leader. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Running, Role: LEADER
I20260929 14:44:42.881564 14740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:42.882099 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:42.883116 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660ab6d618a4f7da73a8bf6869eceb0. Latest consensus state: current_term: 2 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:42.883148 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:42.883241 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:42.883245 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:42.883586 14749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:42.888487 14749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:42.889353 14749 catalog_manager.cc:1259] Loaded cluster ID: b8cd32d365614fd8b45304404b1c4b34
I20260929 14:44:42.889430 14749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:44:42.890940 14749 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 14:44:42.895334 14749 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 14:44:42.901065 14749 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:44:42.901919 14749 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Loaded TSK: 0
I20260929 14:44:42.903231 14749 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:44:42.965991 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:42.966269 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:42.966312 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:42.970474 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:42.970542 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:42.970604 14738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:44:42.975999 14738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42069
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:42.977138 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:42.978158 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:42.988929 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:42.988895 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.989051 14738 server_base.cc:1034] running on GCE node
W20260929 14:44:42.988889 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:42.989670 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:42.990365 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:42.991533 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790693082991515 us; error 30 us; skew 500 ppm
Sep 29 14:44:42 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:42.994356 14738 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:44:42.995277 14738 webserver.cc:460] Webserver started at http://127.0.98.1:42069/ using document root <none> and password file <none>
I20260929 14:44:42.995811 14738 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:42.995910 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:43.000213 14738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:43.002810 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.004195 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:43.004315 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "9c2416daec5faee79374a05e5d67571a"
server_key_iv: "7dffa386034330c164679683dd5e5c19"
server_key_version: "encryptionkey@0"
I20260929 14:44:43.004951 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:43.022023 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:43.025328 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:43.025501 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:43.026116 14738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:43.027338 14780 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:44:43.031530 14738 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:44:43.031615 14738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 14:44:43.031670 14738 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:44:43.033456 14738 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:44:43.033509 14738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260929 14:44:43.033897 14780 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap starting.
I20260929 14:44:43.042897 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44693
I20260929 14:44:43.042966 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44693 every 8 connection(s)
I20260929 14:44:43.043864 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:44:43.044224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14738
I20260929 14:44:43.044709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=46471
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:43.054710 14780 log.cc:824] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:43.061558 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.046109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:36143)
0929 14:44:43.046765 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.046836 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.048023 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.049266 (+ 1243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.049274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.049703 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:43.050264 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.050280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.051558 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.051560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.052425 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.052435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.053177 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.053181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.053370 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.053929 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:43.053946 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:43.055451 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:43.058105 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.058111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.058126 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.058453 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.058885 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.058889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.058890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.059070 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.059759 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:43.059765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:43.060072 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.060264 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.060591 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":159,"threads_started":1}
I20260929 14:44:43.061564 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.046096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36143 (local address 127.0.98.62:40209)
0929 14:44:43.046452 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.046471 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.046960 (+ 489us) server_negotiation.cc:407] Connection header received
0929 14:44:43.048490 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.048503 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.048821 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.049193 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:43.050439 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.051400 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.052547 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.053058 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.053220 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.055695 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:43.055702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:43.055721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:43.055760 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:43.057973 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.058578 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.058585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.058591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.058759 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.059160 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.059163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.059165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.059630 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:43.059744 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.060419 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.060727 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260929 14:44:43.062871 14780 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:43.063288 14780 tablet_bootstrap.cc:492] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap complete.
I20260929 14:44:43.064242 14780 ts_tablet_manager.cc:1397] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.008s
I20260929 14:44:43.064697 14888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
I20260929 14:44:43.064960 14888 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:43.065507 14888 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:43.067106 14780 raft_consensus.cc:348] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:43.067596 14780 raft_consensus.cc:729] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf608b45f327438ca6a3931b1a8cdc5f, State: Initialized, Role: FOLLOWER
I20260929 14:44:43.068048 14780 consensus_queue.cc:260] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } }
I20260929 14:44:43.068480 14700 ts_manager.cc:194] Registered new tserver with Master: cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
I20260929 14:44:43.069013 14780 ts_tablet_manager.cc:1428] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 14:44:43.073026 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36143
I20260929 14:44:43.075907 14888 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:44:43.083212 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.072023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:38762)
0929 14:44:43.072428 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.072476 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.072607 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.073527 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.073532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.073926 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:43.074038 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.074045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.075335 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.075339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.076257 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.076271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.076961 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.076981 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.077100 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.077711 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:43.077738 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:43.079210 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:43.081287 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.081297 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.081303 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.081615 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.081935 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.081938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.081939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.082022 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.082514 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:43.082520 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:43.082640 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.082799 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.083027 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260929 14:44:43.083212 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.072100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.0.98.1:44693)
0929 14:44:43.072837 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.072846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.072895 (+ 49us) server_negotiation.cc:407] Connection header received
0929 14:44:43.072966 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.072970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.073127 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.073362 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:43.074155 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.075194 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.076396 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.076822 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.076965 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.079392 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:43.079401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:43.079411 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:43.079437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:43.081169 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.081729 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.081734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.081735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.081817 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.082104 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.082107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.082108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.082352 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:43.082514 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.082880 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.083037 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":650,"thread_start_us":101,"threads_started":1}
I20260929 14:44:43.084345 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e5943adec224c4eb646b0b6c7d917ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38762
I20260929 14:44:43.085326 14897 tablet_replica.cc:330] stopping tablet replica
I20260929 14:44:43.085661 14897 raft_consensus.cc:2229] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:43.085918 14897 raft_consensus.cc:2258] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:43.087009 14897 ts_tablet_manager.cc:1905] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:43.091753 14897 ts_tablet_manager.cc:1918] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:43.091873 14897 log.cc:1196] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/9e5943adec224c4eb646b0b6c7d917ce
I20260929 14:44:43.092139 14897 ts_tablet_manager.cc:1939] T 9e5943adec224c4eb646b0b6c7d917ce P cf608b45f327438ca6a3931b1a8cdc5f: Deleting consensus metadata
I20260929 14:44:43.094244 14687 catalog_manager.cc:4841] TS cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693): tablet 9e5943adec224c4eb646b0b6c7d917ce (table test-table [id=9b8bca9f3e7349f0a2ac46f28bd3e9a0]) successfully deleted
W20260929 14:44:43.173085 14890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:43.173379 14890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:43.173425 14890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:43.177511 14890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:43.177580 14890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:43.177642 14890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:44:43.183065 14890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46471
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:43.184144 14890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:43.185150 14890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:43.194620 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:43.194620 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:43.194654 14890 server_base.cc:1034] running on GCE node
W20260929 14:44:43.194628 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:43.195200 14890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:43.195820 14890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:43.196998 14890 hybrid_clock.cc:648] HybridClock initialized: now 1790693083196959 us; error 45 us; skew 500 ppm
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:43.200166 14890 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:44:43.201211 14890 webserver.cc:460] Webserver started at http://127.0.98.2:46471/ using document root <none> and password file <none>
I20260929 14:44:43.201769 14890 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:43.201843 14890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:43.206022 14890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:44:43.208077 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.209571 14890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:43.209685 14890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "9ad9361c9ae44b66b60d40cf53e38629"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "29e95d977dbc1cc21d0473244023e3a3"
server_key_iv: "e9d53c2b025dc73b28ea939183cd3deb"
server_key_version: "encryptionkey@0"
I20260929 14:44:43.210256 14890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:43.220309 14890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:43.223284 14890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:43.223457 14890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:43.224032 14890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:43.224834 14890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:43.224920 14890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.224985 14890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:43.225013 14890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.233460 14890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44899
I20260929 14:44:43.233486 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44899 every 8 connection(s)
I20260929 14:44:43.234504 14890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:44:43.243149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 14890
I20260929 14:44:43.243636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33887
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=46365
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:43.249015 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.236484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48067 (local address 127.0.98.62:40209)
0929 14:44:43.236612 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.236615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.237494 (+ 879us) server_negotiation.cc:407] Connection header received
0929 14:44:43.238912 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.238915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.238974 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.239047 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:43.240334 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.240881 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.242035 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.242264 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.242353 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.245136 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:43.245139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:43.245141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:43.245169 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:43.246525 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.247170 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.247173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.247175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.247280 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.247713 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.247716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.247717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.247926 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:43.248004 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.248686 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.248809 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:44:43.250219 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.236778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:48067)
0929 14:44:43.237323 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.237368 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.238678 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.239261 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.239270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.239656 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:43.240168 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.240182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.240997 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.240999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.241907 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.241917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.242409 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.242413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.242586 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.243161 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:43.243177 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:43.244958 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:43.246650 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.246656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.246673 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.247039 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.247406 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.247409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.247411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.247612 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.248026 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:43.248032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:43.248400 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.248642 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.249068 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":112,"threads_started":1}
I20260929 14:44:43.251621 15026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
I20260929 14:44:43.251858 15026 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:43.252385 15026 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:43.253538 14700 ts_manager.cc:194] Registered new tserver with Master: 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899)
I20260929 14:44:43.254402 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48067
W20260929 14:44:43.357896 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:43.358196 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:43.358238 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:43.362323 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:43.362385 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:43.362443 15029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:44:43.367894 15029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46365
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:43.369041 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:43.370080 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:43.379985 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:43.379993 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:43.379987 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:43.380167 15029 server_base.cc:1034] running on GCE node
I20260929 14:44:43.380677 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:43.381347 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:43.382525 15029 hybrid_clock.cc:648] HybridClock initialized: now 1790693083382497 us; error 39 us; skew 500 ppm
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:43.385434 15029 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:44:43.386381 15029 webserver.cc:460] Webserver started at http://127.0.98.3:46365/ using document root <none> and password file <none>
I20260929 14:44:43.386926 15029 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:43.387008 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:43.391088 15029 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:44:43.393318 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.394662 15029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:44:43.394778 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "61bbd520b21f4a24b5390a6db1e0a98c"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "882830a1c42012f6fb62c99dd3f503e1"
server_key_iv: "93ab3adc0fcfdd693107f3b99a0216cc"
server_key_version: "encryptionkey@0"
I20260929 14:44:43.395327 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:43.423234 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:43.426620 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:43.426810 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:43.427433 15029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:43.428275 15029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:43.428325 15029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.428388 15029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:43.428416 15029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.437312 15029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33887
I20260929 14:44:43.437359 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33887 every 8 connection(s)
I20260929 14:44:43.438377 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:44:43.440634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15029
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:44:43.450109 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.441322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.0.98.62:40209)
0929 14:44:43.441561 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.441566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.441824 (+ 258us) server_negotiation.cc:407] Connection header received
0929 14:44:43.441949 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.441952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.441999 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.442091 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:43.442793 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.443581 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.444584 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.444814 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.444948 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.446681 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:43.446684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:43.446686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:43.446713 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:43.448344 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.448787 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.448790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.448792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.448903 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.449162 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.449166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.449167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.449344 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:43.449613 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.449777 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.449943 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260929 14:44:43.452885 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.440359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41483 (local address 127.0.98.62:40209)
0929 14:44:43.440537 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.440542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.441754 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 14:44:43.442724 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.442728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.442777 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.442849 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:43.444228 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.444898 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.446270 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.446661 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.446737 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.449358 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:43.449361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:43.449363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:43.449383 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:43.450848 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.451409 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.451413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.451414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.451488 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:43.451804 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:43.451807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:43.451809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:43.451974 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:43.452056 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.452601 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.452713 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 14:44:43.453739 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.440712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:41483)
0929 14:44:43.441583 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.441640 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.442516 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.443055 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.443064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.443529 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:43.444074 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.444089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.445007 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.445011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.446132 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.446142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.446805 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.446809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.447053 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.447478 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:43.447498 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:43.449205 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:43.450951 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.450958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.450976 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.451317 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.451567 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:43.451570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:43.451572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:43.451720 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:43.452070 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:43.452077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:43.452373 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.452547 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.452917 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":159,"threads_started":1}
I20260929 14:44:43.454909 15158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40209
I20260929 14:44:43.455132 15158 heartbeater.cc:464] Registering TS with master...
I20260929 14:44:43.455639 15158 heartbeater.cc:511] Master 127.0.98.62:40209 requested a full tablet report, sending...
I20260929 14:44:43.456744 14700 ts_manager.cc:194] Registered new tserver with Master: 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887)
I20260929 14:44:43.457476 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41483
I20260929 14:44:43.464495 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:44:43.468940 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.465183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.0.98.62:40209)
0929 14:44:43.465332 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.465336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.465553 (+ 217us) server_negotiation.cc:407] Connection header received
0929 14:44:43.465719 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.465721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.465830 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.465916 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:44:43.466184 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.466682 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.467564 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.467767 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.467839 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.468132 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:44:43.468552 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:44:43.468652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.468716 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.468763 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:44:43.469678 14700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:44:43.473131 14700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:44:43.481148 14822 tablet_service.cc:1467] Processing CreateTablet for tablet cd9df919f6644cca97edfce825c6ad8f (DEFAULT_TABLE table=test-table [id=70835c9731ce46d59da0a4352e205880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:43.481899 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9df919f6644cca97edfce825c6ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:43.484788 14780 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap starting.
I20260929 14:44:43.485690 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.480117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.0.1:59598)
0929 14:44:43.480235 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.480250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.480330 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.481425 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.481428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.481447 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:43.481555 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.481561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.483078 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.483081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.484202 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.484209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.485234 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.485237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.485336 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.485396 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.485461 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.485552 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260929 14:44:43.485739 14780 tablet_bootstrap.cc:654] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:43.486008 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.480403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33887 (local address 127.0.0.1:46056)
0929 14:44:43.480766 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.480778 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.480888 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.481589 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.481592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.481610 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:43.481743 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.481750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.482842 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.482845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.484138 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.484149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.485458 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.485461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.485542 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.485560 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.485618 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.485674 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":82,"threads_started":1}
I20260929 14:44:43.486193 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.480320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.0.98.2:44899)
0929 14:44:43.480902 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.480918 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.480943 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:44:43.481021 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.481026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.481251 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.481703 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:43.481868 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.482951 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.484316 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.485127 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.485234 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.485496 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.485555 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.485653 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":157,"threads_started":1}
I20260929 14:44:43.486261 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.480779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.0.98.3:33887)
0929 14:44:43.481154 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.481166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.481187 (+ 21us) server_negotiation.cc:407] Connection header received
0929 14:44:43.481261 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.481265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.481457 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.481605 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:43.481842 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.482705 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.484284 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.485130 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.485236 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.485387 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.485762 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.485872 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":182,"threads_started":1}
I20260929 14:44:43.487308 14780 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: No bootstrap required, opened a new log
I20260929 14:44:43.487403 14780 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 14:44:43.487799 14780 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.487903 14780 raft_consensus.cc:374] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:43.487939 14780 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf608b45f327438ca6a3931b1a8cdc5f, State: Initialized, Role: FOLLOWER
I20260929 14:44:43.488061 14780 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.488430 14780 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260929 14:44:43.489327 14960 tablet_service.cc:1467] Processing CreateTablet for tablet cd9df919f6644cca97edfce825c6ad8f (DEFAULT_TABLE table=test-table [id=70835c9731ce46d59da0a4352e205880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:43.489327 15092 tablet_service.cc:1467] Processing CreateTablet for tablet cd9df919f6644cca97edfce825c6ad8f (DEFAULT_TABLE table=test-table [id=70835c9731ce46d59da0a4352e205880]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:44:43.490505 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9df919f6644cca97edfce825c6ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:43.490499 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9df919f6644cca97edfce825c6ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:43.498966 15172 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Bootstrap starting.
I20260929 14:44:43.499599 15171 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Bootstrap starting.
I20260929 14:44:43.502120 15171 tablet_bootstrap.cc:654] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:43.502502 15172 tablet_bootstrap.cc:654] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Neither blocks nor log segments found. Creating new log.
I20260929 14:44:43.503373 15171 log.cc:824] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:43.503373 15172 log.cc:824] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:43.505376 15172 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: No bootstrap required, opened a new log
I20260929 14:44:43.505379 15171 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: No bootstrap required, opened a new log
I20260929 14:44:43.505580 15171 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:44:43.505580 15172 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 14:44:43.508580 15172 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.508580 15171 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.508826 15171 raft_consensus.cc:374] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:43.508832 15172 raft_consensus.cc:374] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:44:43.508895 15171 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bbd520b21f4a24b5390a6db1e0a98c, State: Initialized, Role: FOLLOWER
I20260929 14:44:43.508913 15172 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad9361c9ae44b66b60d40cf53e38629, State: Initialized, Role: FOLLOWER
I20260929 14:44:43.509364 15172 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.509366 15171 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.510003 15158 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
I20260929 14:44:43.510004 15026 heartbeater.cc:503] Master 127.0.98.62:40209 was elected leader, sending a full tablet report...
I20260929 14:44:43.510396 15172 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260929 14:44:43.510397 15171 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260929 14:44:43.545346 14889 tablet.cc:2369] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:44:43.553723 15175 raft_consensus.cc:482] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:44:43.553982 15175 raft_consensus.cc:504] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.555184 15175 leader_election.cc:288] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899), cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
I20260929 14:44:43.559129 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.555456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.98.3:48655)
0929 14:44:43.555647 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.555663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.555772 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.556057 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.556060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.556075 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:43.556197 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.556204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.557076 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.557080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.557942 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.557952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.558671 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.558787 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.558814 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.558900 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.558988 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 14:44:43.559165 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.555607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48655 (local address 127.0.98.2:44899)
0929 14:44:43.555845 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.555850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.555864 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:44:43.555920 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.555923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.555971 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.556040 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:43.556404 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.556967 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558076 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.558547 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558627 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.558680 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.558955 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.559037 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 14:44:43.559381 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.555481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.98.3:48129)
0929 14:44:43.555783 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.555796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:43.555915 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:43.556177 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:43.556179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:43.556207 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:44:43.556314 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.556321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.557262 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.557264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.558142 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:43.558153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.558964 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:43.559069 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.559092 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:44:43.559182 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 14:44:43.559249 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":124,"threads_started":1}
I20260929 14:44:43.559810 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.555599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48129 (local address 127.0.98.1:44693)
0929 14:44:43.555746 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.555750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.555885 (+ 135us) server_negotiation.cc:407] Connection header received
0929 14:44:43.555996 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.555999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.556059 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.556157 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:44:43.556496 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.557159 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558274 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.558826 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.558926 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.559091 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.559648 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.559720 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:44:43.559985 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad9361c9ae44b66b60d40cf53e38629" is_pre_election: true
I20260929 14:44:43.560333 14980 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 0.
I20260929 14:44:43.560282 14842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" is_pre_election: true
I20260929 14:44:43.560563 14842 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 0.
I20260929 14:44:43.560887 15046 leader_election.cc:302] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bbd520b21f4a24b5390a6db1e0a98c, 9ad9361c9ae44b66b60d40cf53e38629; no voters:
I20260929 14:44:43.561189 15175 raft_consensus.cc:2781] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:44:43.561269 15175 raft_consensus.cc:482] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:44:43.561300 15175 raft_consensus.cc:3037] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:43.562307 15175 raft_consensus.cc:504] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.562659 15175 leader_election.cc:288] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 election: Requested vote from peers 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899), cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
I20260929 14:44:43.563086 14842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
I20260929 14:44:43.563086 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad9361c9ae44b66b60d40cf53e38629"
I20260929 14:44:43.563230 14842 raft_consensus.cc:3037] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:43.563228 14980 raft_consensus.cc:3037] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:44:43.564535 14842 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 1.
I20260929 14:44:43.564535 14980 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61bbd520b21f4a24b5390a6db1e0a98c in term 1.
I20260929 14:44:43.564850 15048 leader_election.cc:302] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61bbd520b21f4a24b5390a6db1e0a98c, cf608b45f327438ca6a3931b1a8cdc5f; no voters:
I20260929 14:44:43.565044 15175 raft_consensus.cc:2781] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:44:43.565277 15175 raft_consensus.cc:686] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 LEADER]: Becoming Leader. State: Replica: 61bbd520b21f4a24b5390a6db1e0a98c, State: Running, Role: LEADER
I20260929 14:44:43.565645 15175 consensus_queue.cc:237] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:43.569104 14700 catalog_manager.cc:5495] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c reported cstate change: term changed from 0 to 1, leader changed from <none> to 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } health_report { overall_health: UNKNOWN } } }
I20260929 14:44:43.589339 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.585473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.0.98.3:33887)
0929 14:44:43.585609 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:43.585613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:43.585685 (+ 72us) server_negotiation.cc:407] Connection header received
0929 14:44:43.585863 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:43.585866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:43.585931 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:43.586022 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:44:43.586248 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.586897 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.587837 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:43.588049 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:43.588147 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:43.588438 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:44:43.588895 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:44:43.589018 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:43.589054 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 14:44:43.589114 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:44:43.595554 14980 raft_consensus.cc:1261] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Refusing update from remote peer 61bbd520b21f4a24b5390a6db1e0a98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:43.595549 14842 raft_consensus.cc:1261] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Refusing update from remote peer 61bbd520b21f4a24b5390a6db1e0a98c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:44:43.596235 15175 consensus_queue.cc:1035] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:43.596457 15178 consensus_queue.cc:1035] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:44:43.603003 15180 mvcc.cc:205] Tried to move back new op lower bound from 7334678870400200704 to 7334678870287482880. Current Snapshot: MvccSnapshot[applied={T|T < 7334678870400200704}]
I20260929 14:44:43.603066 15181 mvcc.cc:205] Tried to move back new op lower bound from 7334678870400200704 to 7334678870287482880. Current Snapshot: MvccSnapshot[applied={T|T < 7334678870400200704}]
I20260929 14:44:43.612200 14698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:43.612393 14698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:44:43.614969 14698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Sending DeleteTablet for 3 replicas of tablet cd9df919f6644cca97edfce825c6ad8f
I20260929 14:44:43.615684 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9df919f6644cca97edfce825c6ad8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38762
I20260929 14:44:43.615772 15092 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9df919f6644cca97edfce825c6ad8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46056
I20260929 14:44:43.615937 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9df919f6644cca97edfce825c6ad8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59598
I20260929 14:44:43.616065 15189 tablet_replica.cc:330] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: stopping tablet replica
I20260929 14:44:43.616199 15189 raft_consensus.cc:2229] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:43.616349 15189 raft_consensus.cc:2258] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:43.616379 15190 tablet_replica.cc:330] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: stopping tablet replica
I20260929 14:44:43.616539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14738
I20260929 14:44:43.616751 15190 raft_consensus.cc:2229] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:44:43.617537 15190 raft_consensus.cc:2258] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:43.618839 15190 ts_tablet_manager.cc:1905] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:44:43.621145 15191 tablet_replica.cc:330] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: stopping tablet replica
I20260929 14:44:43.621200 15190 ts_tablet_manager.cc:1918] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:44:43.621292 15190 log.cc:1196] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/cd9df919f6644cca97edfce825c6ad8f
I20260929 14:44:43.621626 15190 ts_tablet_manager.cc:1939] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Deleting consensus metadata
I20260929 14:44:43.621659 15191 raft_consensus.cc:2229] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:44:43.622128 15191 raft_consensus.cc:2258] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:44:43.622920 14685 catalog_manager.cc:4841] TS 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887): tablet cd9df919f6644cca97edfce825c6ad8f (table test-table [id=70835c9731ce46d59da0a4352e205880]) successfully deleted
W20260929 14:44:43.624764 14687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44693 (error 108)
I20260929 14:44:43.625746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14890
I20260929 14:44:43.626088 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.625741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.0.1:38772)
0929 14:44:43.625931 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.626002 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 14:44:43.626802 14687 catalog_manager.cc:4584] TS cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd9df919f6644cca97edfce825c6ad8f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44693: connect: Connection refused (error 111)
I20260929 14:44:43.631657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15029
I20260929 14:44:43.633200 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:43.631839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.0.1:59606)
0929 14:44:43.633037 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:43.633103 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1104}
I20260929 14:44:43.636809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 14667
I20260929 14:44:43.642705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--webserver_interface=127.0.98.62
--webserver_port=33533
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:44:43.755352 15193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:43.755646 15193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:43.755686 15193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:43.759819 15193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:44:43.759897 15193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:43.759917 15193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:43.759932 15193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:44:43.759946 15193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:44:43.765543 15193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:43.766707 15193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:43.767786 15193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:43.776989 15199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:43.777040 15201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:43.776989 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:43.777155 15193 server_base.cc:1034] running on GCE node
I20260929 14:44:43.777802 15193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:43.779016 15193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:43.780192 15193 hybrid_clock.cc:648] HybridClock initialized: now 1790693083780162 us; error 44 us; skew 500 ppm
Sep 29 14:44:43 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:43.783248 15193 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:44:43.784240 15193 webserver.cc:460] Webserver started at http://127.0.98.62:33533/ using document root <none> and password file <none>
I20260929 14:44:43.784823 15193 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:43.784933 15193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:43.789018 15193 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:44:43.791446 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:43.792855 15193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:43.793040 15193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "2660ab6d618a4f7da73a8bf6869eceb0"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "f33853993d35cedbcdb9cebc58e17f0a"
server_key_iv: "6366119caea38cedeea204bb78643f3d"
server_key_version: "encryptionkey@0"
I20260929 14:44:43.793705 15193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:43.812412 15193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:43.815869 15193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:43.816057 15193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:43.823346 15193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40209
I20260929 14:44:43.823660 15260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40209 every 8 connection(s)
I20260929 14:44:43.824470 15193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:44:43.828924 15261 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:44:43.829368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15193
I20260929 14:44:43.829793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44693
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=42069
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:43.830744 15261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap starting.
I20260929 14:44:43.848045 15261 log.cc:824] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:43.859719 15261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:43.860119 15261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0: Bootstrap complete.
I20260929 14:44:43.864037 15261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:43.864437 15261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Initialized, Role: FOLLOWER
I20260929 14:44:43.864940 15261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:43.865067 15261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:44:43.865139 15261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:44:43.865237 15261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:44:43.867203 15261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:43.867549 15261 leader_election.cc:302] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2660ab6d618a4f7da73a8bf6869eceb0; no voters:
I20260929 14:44:43.867861 15261 leader_election.cc:288] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:44:43.868001 15265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:44:43.868196 15265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [term 3 LEADER]: Becoming Leader. State: Replica: 2660ab6d618a4f7da73a8bf6869eceb0, State: Running, Role: LEADER
I20260929 14:44:43.868499 15265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } }
I20260929 14:44:43.869190 15261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:44:43.869913 15266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:43.870059 15266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:43.870375 15267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2660ab6d618a4f7da73a8bf6869eceb0. Latest consensus state: current_term: 3 leader_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2660ab6d618a4f7da73a8bf6869eceb0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40209 } } }
I20260929 14:44:43.870465 15267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2660ab6d618a4f7da73a8bf6869eceb0 [sys.catalog]: This master's current role is: LEADER
I20260929 14:44:43.870882 15272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:44:43.876207 15272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:44:43.876911 15272 catalog_manager.cc:1259] Loaded cluster ID: b8cd32d365614fd8b45304404b1c4b34
I20260929 14:44:43.876983 15272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 14:44:43.878051 15272 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f56e95c19ad google::LogMessage::Fail() at ??:0
@ 0x7f56e95c488a google::LogMessage::SendToLog() at ??:0
@ 0x7f56e95c14de google::LogMessage::Flush() at ??:0
@ 0x7f56e95c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f56f3f00fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f56f3f01d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f56f3efe579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f56f3f42640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f56eb8133d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f56eb813c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f56eb8153f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f56eb804512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f56ece486db start_thread at ??:0
@ 0x7f56e80bf61f clone at ??:0
W20260929 14:44:43.975142 15263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:43.975594 15263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:43.975668 15263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:43.981937 15263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:43.982036 15263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:43.982127 15263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:44:43.990473 15263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42069
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:43.991811 15263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:43.993108 15263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:44.005582 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:44.005609 15302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:44.005702 15303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:44.005803 15263 server_base.cc:1034] running on GCE node
I20260929 14:44:44.006446 15263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:44.007167 15263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:44.008364 15263 hybrid_clock.cc:648] HybridClock initialized: now 1790693084008327 us; error 48 us; skew 500 ppm
Sep 29 14:44:44 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:44.011829 15263 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:44:44.013021 15263 webserver.cc:460] Webserver started at http://127.0.98.1:42069/ using document root <none> and password file <none>
I20260929 14:44:44.013646 15263 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:44.013724 15263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:44.018738 15263 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 14:44:44.022131 15312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:44.023900 15263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:44:44.024072 15263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
format_stamp: "Formatted at 2026-09-29 14:44:41 on dist-test-slave-g8hp"
server_key: "9c2416daec5faee79374a05e5d67571a"
server_key_iv: "7dffa386034330c164679683dd5e5c19"
server_key_version: "encryptionkey@0"
I20260929 14:44:44.024926 15263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:44.043500 15263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:44.048195 15263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:44.048426 15263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:44.049284 15263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:44.050832 15319 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:44:44.055105 15263 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:44:44.055207 15263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 14:44:44.055277 15263 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:44:44.057614 15263 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:44:44.057673 15263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260929 14:44:44.058142 15319 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap starting.
I20260929 14:44:44.069212 15263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44693
I20260929 14:44:44.069267 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44693 every 8 connection(s)
I20260929 14:44:44.070434 15263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:44:44.078024 15319 log.cc:824] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:44.078462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15263
I20260929 14:44:44.078977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44899
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=46471
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:44.086601 15319 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:44.087054 15319 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Bootstrap complete.
I20260929 14:44:44.088433 15319 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.000s
I20260929 14:44:44.092176 15319 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:44.092805 15319 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf608b45f327438ca6a3931b1a8cdc5f, State: Initialized, Role: FOLLOWER
I20260929 14:44:44.093472 15319 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:44.094628 15319 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f: Time spent starting tablet: real 0.006s user 0.009s sys 0.000s
W20260929 14:44:44.132010 15429 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:44:44.072952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:52497)
0929 14:44:44.073596 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.073662 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:44.074876 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:44.131143 (+ 56267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":163,"threads_started":1}
W20260929 14:44:44.132580 15427 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40209 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40209: Client connection negotiation failed: client connection to 127.0.98.62:40209: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:44:44.133322 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.133077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:39255)
0929 14:44:44.133195 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.133261 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:44:44.133941 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.133755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:55561)
0929 14:44:44.133849 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.133891 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 14:44:44.134096 15427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 14:44:44.212991 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:44.213331 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:44.213459 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:44.217844 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:44.217928 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:44.218012 15431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:44:44.223682 15431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46471
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:44.224895 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:44.225980 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:44:44.236197 15440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:44.236256 15431 server_base.cc:1034] running on GCE node
W20260929 14:44:44.236212 15438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:44.236212 15437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:44.236887 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:44.237579 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:44.238803 15431 hybrid_clock.cc:648] HybridClock initialized: now 1790693084238782 us; error 35 us; skew 500 ppm
Sep 29 14:44:44 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:44.242180 15431 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:44:44.243287 15431 webserver.cc:460] Webserver started at http://127.0.98.2:46471/ using document root <none> and password file <none>
I20260929 14:44:44.243867 15431 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:44.243943 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:44.248253 15431 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:44.250856 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:44.252372 15431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:44.252511 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "9ad9361c9ae44b66b60d40cf53e38629"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "29e95d977dbc1cc21d0473244023e3a3"
server_key_iv: "e9d53c2b025dc73b28ea939183cd3deb"
server_key_version: "encryptionkey@0"
I20260929 14:44:44.253180 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:44.268077 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:44.271971 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:44.272185 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:44.272825 15431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:44.274044 15454 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:44:44.278431 15431 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:44:44.278538 15431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 14:44:44.278607 15431 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:44:44.280537 15431 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:44:44.280591 15431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260929 14:44:44.280910 15454 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Bootstrap starting.
I20260929 14:44:44.286754 15454 log.cc:824] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:44.290513 15431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44899
I20260929 14:44:44.290616 15562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44899 every 8 connection(s)
I20260929 14:44:44.291419 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:44:44.295059 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.293578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:33591)
0929 14:44:44.294141 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.294281 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":129,"threads_started":1}
W20260929 14:44:44.295616 15563 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40209 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40209: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
I20260929 14:44:44.295709 15454 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:44.296075 15454 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Bootstrap complete.
I20260929 14:44:44.296294 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.296058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:60225)
0929 14:44:44.296184 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.296237 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 14:44:44.296443 15454 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20260929 14:44:44.296978 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.296722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:53401)
0929 14:44:44.296830 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.296916 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 14:44:44.297163 15563 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:44:44.299372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15431
I20260929 14:44:44.299762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33887
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40209
--webserver_port=46365
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:44:44.299383 15454 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:44.299865 15454 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad9361c9ae44b66b60d40cf53e38629, State: Initialized, Role: FOLLOWER
I20260929 14:44:44.300333 15454 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:44.301370 15454 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 14:44:44.301568 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.301289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:53639)
0929 14:44:44.301437 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.301497 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:44:44.302156 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.301952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:48893)
0929 14:44:44.302049 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.302104 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 14:44:44.413635 15567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:44:44.413901 15567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:44:44.413949 15567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:44:44.418001 15567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:44:44.418071 15567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:44:44.418133 15567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:44:44.423540 15567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46365
--tserver_master_addrs=127.0.98.62:40209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:44:44.424628 15567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:44:44.425711 15567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:44:44.435928 15567 server_base.cc:1034] running on GCE node
W20260929 14:44:44.435900 15572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:44.436121 15575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:44:44.436054 15573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:44:44.436576 15567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:44:44.437342 15567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:44:44.438511 15567 hybrid_clock.cc:648] HybridClock initialized: now 1790693084438491 us; error 34 us; skew 500 ppm
Sep 29 14:44:44 dist-test-slave-g8hp krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:44:44.441288 15567 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:44:44.442178 15567 webserver.cc:460] Webserver started at http://127.0.98.3:46365/ using document root <none> and password file <none>
I20260929 14:44:44.442723 15567 fs_manager.cc:359] Metadata directory not provided
I20260929 14:44:44.442785 15567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:44:44.446779 15567 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:44:44.448958 15582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:44.450353 15567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:44:44.450471 15567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "61bbd520b21f4a24b5390a6db1e0a98c"
format_stamp: "Formatted at 2026-09-29 14:44:42 on dist-test-slave-g8hp"
server_key: "882830a1c42012f6fb62c99dd3f503e1"
server_key_iv: "93ab3adc0fcfdd693107f3b99a0216cc"
server_key_version: "encryptionkey@0"
I20260929 14:44:44.451047 15567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:44:44.463487 15567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:44:44.466694 15567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:44:44.466873 15567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:44:44.467458 15567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:44:44.468272 15567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:44:44.468318 15567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:44.468369 15567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:44:44.468397 15567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:44:44.476992 15567 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33887
I20260929 14:44:44.477023 15695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33887 every 8 connection(s)
I20260929 14:44:44.478010 15567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:44:44.481907 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.480305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:43539)
0929 14:44:44.481044 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.481158 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":121,"threads_started":1}
W20260929 14:44:44.482378 15696 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40209 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40209: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
I20260929 14:44:44.483001 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.482787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:40569)
0929 14:44:44.482896 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.482945 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:44:44.483664 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:44.483374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:44841)
0929 14:44:44.483567 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:44.483615 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:44:44.483814 15696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:44:44.486753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15567
I20260929 14:44:45.135320 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.134816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:55291)
0929 14:44:45.135141 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.135207 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260929 14:44:45.303375 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.302903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:56961)
0929 14:44:45.303186 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.303254 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260929 14:44:45.484817 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.484382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:46435)
0929 14:44:45.484646 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.484708 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260929 14:44:45.636479 15703 raft_consensus.cc:482] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:44:45.636709 15703 raft_consensus.cc:504] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:45.637953 15703 leader_election.cc:288] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887), cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
Sep 29 14:44:45 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 14:44:45 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:44:45.654909 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.638412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44941 (local address 127.0.98.3:33887)
0929 14:44:45.638847 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:45.638872 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:45.638917 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:44:45.640381 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:45.640390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:45.640788 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:45.641146 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:45.641913 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.643091 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.644740 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.645480 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.645614 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:45.648907 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:45.648915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:45.648932 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:45.648967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:45.651187 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652079 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:45.652086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:45.652093 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:45.652272 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652752 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:45.652756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:45.652758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:45.653409 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:45.653546 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:45.654387 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 14:44:45.654697 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":98,"threads_started":1}
I20260929 14:44:45.654904 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.638442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53615 (local address 127.0.98.1:44693)
0929 14:44:45.638727 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:45.638736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:45.638886 (+ 150us) server_negotiation.cc:407] Connection header received
0929 14:44:45.639634 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:45.639642 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:45.639906 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:45.640103 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:45.642185 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.643878 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.644971 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.645632 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.645799 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:45.649134 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:45.649142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:45.649159 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:45.649195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:45.651200 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652093 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:45.652099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:45.652103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:45.652272 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652794 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:45.652798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:45.652799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:45.653409 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:45.653546 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:45.654387 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 14:44:45.654701 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260929 14:44:45.654913 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.638196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33887 (local address 127.0.98.2:44941)
0929 14:44:45.638633 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.638662 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:45.639303 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:45.641112 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:45.641114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:45.641284 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:45.641657 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:45.641678 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.643289 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.643293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:45.644592 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:45.644603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.645674 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.645678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:45.645932 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:45.646747 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:45.646770 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:45.648615 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:45.651362 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:45.651371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:45.651390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:45.651928 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:45.652385 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:45.652390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:45.652638 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:45.653571 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:45.653579 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:45.653993 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 14:44:45.654219 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 14:44:45.654739 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":244,"spinlock_wait_cycles":32640}
I20260929 14:44:45.654953 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.638299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44693 (local address 127.0.98.2:53615)
0929 14:44:45.638773 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.638787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:45.639164 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:45.640315 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:45.640325 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:45.640909 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:45.641657 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:45.641889 (+ 232us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.644068 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.644079 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:45.644815 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:45.644826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:45.645769 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:45.645773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:45.646074 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:45.646829 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:45.646841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:45.648950 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:45.651362 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:45.651371 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:45.651390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:45.651928 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:45.652385 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:45.652388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:45.652390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:45.652694 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:45.653599 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:45.653602 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:45.653993 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 14:44:45.654219 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 14:44:45.654685 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":269,"spinlock_wait_cycles":59520,"thread_start_us":83,"threads_started":1}
I20260929 14:44:45.656309 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "9ad9361c9ae44b66b60d40cf53e38629" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" is_pre_election: true
I20260929 14:44:45.656689 15381 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad9361c9ae44b66b60d40cf53e38629 in term 1.
I20260929 14:44:45.657968 15451 leader_election.cc:302] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad9361c9ae44b66b60d40cf53e38629, cf608b45f327438ca6a3931b1a8cdc5f; no voters:
W20260929 14:44:45.658185 15449 leader_election.cc:341] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887): Not found: Tablet not found: cd9df919f6644cca97edfce825c6ad8f
I20260929 14:44:45.658316 15703 raft_consensus.cc:2781] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:44:45.658425 15703 raft_consensus.cc:482] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:44:45.658538 15703 raft_consensus.cc:3037] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:44:45.660832 15703 raft_consensus.cc:504] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:45.661271 15703 leader_election.cc:288] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 election: Requested vote from peers 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887), cf608b45f327438ca6a3931b1a8cdc5f (127.0.98.1:44693)
I20260929 14:44:45.661834 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9df919f6644cca97edfce825c6ad8f" candidate_uuid: "9ad9361c9ae44b66b60d40cf53e38629" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cf608b45f327438ca6a3931b1a8cdc5f"
I20260929 14:44:45.662052 15381 raft_consensus.cc:3037] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:44:45.662190 15449 leader_election.cc:341] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887): Not found: Tablet not found: cd9df919f6644cca97edfce825c6ad8f
I20260929 14:44:45.664333 15381 raft_consensus.cc:2454] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad9361c9ae44b66b60d40cf53e38629 in term 2.
I20260929 14:44:45.664760 15451 leader_election.cc:302] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ad9361c9ae44b66b60d40cf53e38629, cf608b45f327438ca6a3931b1a8cdc5f; no voters: 61bbd520b21f4a24b5390a6db1e0a98c
I20260929 14:44:45.664987 15703 raft_consensus.cc:2781] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:44:45.665174 15703 raft_consensus.cc:686] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [term 2 LEADER]: Becoming Leader. State: Replica: 9ad9361c9ae44b66b60d40cf53e38629, State: Running, Role: LEADER
I20260929 14:44:45.665521 15703 consensus_queue.cc:237] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:45.665958 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:45.665665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:35839)
0929 14:44:45.665816 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:45.665880 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:44:46.136631 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.136182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:45245)
0929 14:44:46.136454 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.136521 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
W20260929 14:44:46.222599 15449 consensus_peers.cc:487] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 -> Peer 61bbd520b21f4a24b5390a6db1e0a98c (127.0.98.3:33887): Couldn't send request to peer 61bbd520b21f4a24b5390a6db1e0a98c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd9df919f6644cca97edfce825c6ad8f. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:44:46.224929 15381 raft_consensus.cc:1261] T cd9df919f6644cca97edfce825c6ad8f P cf608b45f327438ca6a3931b1a8cdc5f [term 2 FOLLOWER]: Refusing update from remote peer 9ad9361c9ae44b66b60d40cf53e38629: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:44:46.225703 15710 consensus_queue.cc:1035] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:44:46.227638 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.227373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:50159)
0929 14:44:46.227495 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.227562 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 14:44:46.230194 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.229775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:48385)
0929 14:44:46.230030 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.230092 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":58,"threads_started":1}
I20260929 14:44:46.230350 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.230122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:42121)
0929 14:44:46.230216 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.230274 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 14:44:46.486166 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.485749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:60523)
0929 14:44:46.486003 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.486057 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260929 14:44:46.702136 15718 ts_tablet_manager.cc:927] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Initiating tablet copy from peer 9ad9361c9ae44b66b60d40cf53e38629 (127.0.98.2:44899)
I20260929 14:44:46.702867 15718 tablet_copy_client.cc:316] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:44899
Sep 29 14:44:46 dist-test-slave-g8hp krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 14:44:46.720098 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.703404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44899 (local address 127.0.98.3:34231)
0929 14:44:46.703539 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.703564 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 14:44:46.703687 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:44:46.704338 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:44:46.704344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:44:46.704750 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:44:46.704891 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:46.704898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:46.706050 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:46.706053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:46.706700 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:44:46.706711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:44:46.707318 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:44:46.707321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:44:46.707405 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:46.714588 (+ 7183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:44:46.714608 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:44:46.716225 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:44:46.718245 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:46.718255 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:46.718260 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:46.718497 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:46.718810 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:44:46.718812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:44:46.718813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:44:46.718885 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:44:46.719492 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:44:46.719498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:44:46.719628 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 14:44:46.719769 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 14:44:46.719925 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:44:46.720165 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.703572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34231 (local address 127.0.98.2:44899)
0929 14:44:46.703897 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 14:44:46.703907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:44:46.703927 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:44:46.703971 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:44:46.703975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:44:46.704185 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:44:46.704318 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:44:46.705047 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:46.705870 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:46.706825 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:44:46.707199 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:44:46.707291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:44:46.716432 (+ 9141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:44:46.716449 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:44:46.716466 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:44:46.716502 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:44:46.718118 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:46.718601 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:46.718649 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:46.718650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:46.718740 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:44:46.718999 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:44:46.719002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:44:46.719003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:44:46.719299 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:44:46.719428 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:44:46.719882 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 14:44:46.720028 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":105,"threads_started":1}
I20260929 14:44:46.721411 15537 tablet_copy_service.cc:140] P 9ad9361c9ae44b66b60d40cf53e38629: Received BeginTabletCopySession request for tablet cd9df919f6644cca97edfce825c6ad8f from peer 61bbd520b21f4a24b5390a6db1e0a98c ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34231)
I20260929 14:44:46.721602 15537 tablet_copy_service.cc:161] P 9ad9361c9ae44b66b60d40cf53e38629: Beginning new tablet copy session on tablet cd9df919f6644cca97edfce825c6ad8f from peer 61bbd520b21f4a24b5390a6db1e0a98c at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34231: session id = 61bbd520b21f4a24b5390a6db1e0a98c-cd9df919f6644cca97edfce825c6ad8f
I20260929 14:44:46.722805 15537 tablet_copy_source_session.cc:205] T cd9df919f6644cca97edfce825c6ad8f P 9ad9361c9ae44b66b60d40cf53e38629: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:44:46.726858 15718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9df919f6644cca97edfce825c6ad8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:44:46.733937 15718 tablet_copy_client.cc:791] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: tablet copy: Starting download of 0 data blocks...
I20260929 14:44:46.734134 15718 tablet_copy_client.cc:655] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: tablet copy: Starting download of 1 WAL segments...
I20260929 14:44:46.735992 15718 tablet_copy_client.cc:531] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:44:46.737778 15718 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Bootstrap starting.
I20260929 14:44:46.743808 15718 log.cc:824] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Log is configured to *not* fsync() on all Append() calls
I20260929 14:44:46.751149 15718 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:44:46.751467 15718 tablet_bootstrap.cc:492] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Bootstrap complete.
I20260929 14:44:46.751780 15718 ts_tablet_manager.cc:1397] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260929 14:44:46.754222 15718 raft_consensus.cc:348] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:46.754485 15718 raft_consensus.cc:729] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61bbd520b21f4a24b5390a6db1e0a98c, State: Initialized, Role: FOLLOWER
I20260929 14:44:46.754959 15718 consensus_queue.cc:260] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ad9361c9ae44b66b60d40cf53e38629" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44899 } } peers { permanent_uuid: "61bbd520b21f4a24b5390a6db1e0a98c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33887 } } peers { permanent_uuid: "cf608b45f327438ca6a3931b1a8cdc5f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44693 } }
I20260929 14:44:46.755905 15718 ts_tablet_manager.cc:1428] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260929 14:44:46.756224 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.755932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:51913)
0929 14:44:46.756082 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.756139 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 14:44:46.756675 15537 tablet_copy_service.cc:342] P 9ad9361c9ae44b66b60d40cf53e38629: Request end of tablet copy session 61bbd520b21f4a24b5390a6db1e0a98c-cd9df919f6644cca97edfce825c6ad8f received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34231
I20260929 14:44:46.756804 15537 tablet_copy_service.cc:433] P 9ad9361c9ae44b66b60d40cf53e38629: ending tablet copy session 61bbd520b21f4a24b5390a6db1e0a98c-cd9df919f6644cca97edfce825c6ad8f on tablet cd9df919f6644cca97edfce825c6ad8f with peer 61bbd520b21f4a24b5390a6db1e0a98c
I20260929 14:44:46.756933 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:46.756627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:53249)
0929 14:44:46.756762 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:46.756841 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 14:44:47.231356 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:47.230980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:54691)
0929 14:44:47.231171 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:47.231237 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260929 14:44:47.231452 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:47.231106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:53701)
0929 14:44:47.231307 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:47.231361 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260929 14:44:47.231947 15650 raft_consensus.cc:1203] T cd9df919f6644cca97edfce825c6ad8f P 61bbd520b21f4a24b5390a6db1e0a98c [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 14:44:47.232801 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:47.232534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:48435)
0929 14:44:47.232652 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:47.232708 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 14:44:47.232903 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:47.232611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:51661)
0929 14:44:47.232767 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:47.232813 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 14:44:48.232964 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:48.232494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:57319)
0929 14:44:48.232769 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:48.232834 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
I20260929 14:44:48.233892 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:48.233536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:54671)
0929 14:44:48.233758 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:48.233812 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260929 14:44:48.233930 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:48.233534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:52281)
0929 14:44:48.233817 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:48.233859 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":55,"threads_started":1}
I20260929 14:44:49.234618 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:49.234056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:46777)
0929 14:44:49.234423 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:49.234498 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":102,"threads_started":1}
I20260929 14:44:49.234970 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:49.234585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:57187)
0929 14:44:49.234804 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:49.234864 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260929 14:44:49.235008 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:49.234586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:58677)
0929 14:44:49.234877 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:49.234928 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":72,"threads_started":1}
I20260929 14:44:50.236164 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:50.235513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:60405)
0929 14:44:50.235980 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:50.236047 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":79,"threads_started":1}
I20260929 14:44:50.236326 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:50.235860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:57257)
0929 14:44:50.236183 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:50.236228 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":64,"threads_started":1}
I20260929 14:44:50.236340 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:50.235908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:47583)
0929 14:44:50.236201 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:50.236254 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":76,"threads_started":1}
I20260929 14:44:50.236941 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:50.236709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:52331)
0929 14:44:50.236809 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:50.236852 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 14:44:51.237576 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:51.237047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:50523)
0929 14:44:51.237404 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:51.237465 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
I20260929 14:44:51.237576 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:51.237009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:48097)
0929 14:44:51.237403 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:51.237465 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":95,"threads_started":1}
I20260929 14:44:51.238121 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:51.237777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:49319)
0929 14:44:51.237996 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:51.238045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260929 14:44:52.239024 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:52.238505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:49451)
0929 14:44:52.238858 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:52.238920 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":82,"threads_started":1}
I20260929 14:44:52.239223 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:52.238505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:50657)
0929 14:44:52.239040 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:52.239119 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":80,"threads_started":1}
I20260929 14:44:52.239372 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:52.238829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:35607)
0929 14:44:52.239228 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:52.239285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":101,"threads_started":1}
I20260929 14:44:53.240835 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:53.240148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:50691)
0929 14:44:53.240624 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:53.240716 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":131,"threads_started":1}
I20260929 14:44:53.240847 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:53.240216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:49353)
0929 14:44:53.240624 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:53.240708 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":84,"threads_started":1}
I20260929 14:44:53.240834 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:53.240246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:57827)
0929 14:44:53.240652 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:53.240712 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":116,"threads_started":1}
I20260929 14:44:54.242182 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:54.241675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:44541)
0929 14:44:54.241968 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:54.242038 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260929 14:44:54.242322 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:54.241895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:53959)
0929 14:44:54.242154 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:54.242209 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
I20260929 14:44:54.242897 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:54.242240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:58497)
0929 14:44:54.242712 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:54.242779 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":67,"threads_started":1}
I20260929 14:44:55.243983 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:55.243408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:48139)
0929 14:44:55.243777 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:55.243842 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260929 14:44:55.243991 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:55.243408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:46757)
0929 14:44:55.243715 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:55.243789 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260929 14:44:55.244400 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:55.243644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:44495)
0929 14:44:55.244241 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:55.244306 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":74,"threads_started":1}
I20260929 14:44:56.245296 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:56.244946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:38805)
0929 14:44:56.245140 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:56.245196 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":56,"threads_started":1}
I20260929 14:44:56.245280 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:56.244759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:35501)
0929 14:44:56.245088 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:56.245157 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
I20260929 14:44:56.245749 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:56.245227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:43675)
0929 14:44:56.245598 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:56.245659 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":83,"threads_started":1}
I20260929 14:44:57.246833 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:57.246469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:56947)
0929 14:44:57.246701 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:57.246753 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260929 14:44:57.246833 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:57.246285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:35929)
0929 14:44:57.246629 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:57.246714 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":87,"threads_started":1}
I20260929 14:44:57.246980 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:57.246285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:36677)
0929 14:44:57.246570 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:57.246876 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260929 14:44:58.248117 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:58.247654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:40867)
0929 14:44:58.247927 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:58.247996 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260929 14:44:58.248207 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:58.247700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:50967)
0929 14:44:58.248021 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:58.248101 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":76,"threads_started":1}
I20260929 14:44:58.248239 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:58.247696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:57971)
0929 14:44:58.248100 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:58.248155 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":78,"threads_started":1}
I20260929 14:44:59.249557 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:59.249076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.3:36907)
0929 14:44:59.249378 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:59.249446 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260929 14:44:59.249619 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:59.249189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.2:48567)
0929 14:44:59.249457 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:59.249536 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":68,"threads_started":1}
I20260929 14:44:59.249619 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:44:59.249182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40209 (local address 127.0.98.1:40793)
0929 14:44:59.249440 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:44:59.249503 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":55,"threads_started":1}
I20260929 14:44:59.497042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15263
I20260929 14:44:59.502214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15431
I20260929 14:44:59.507256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15567
2026-09-29T14:44:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19027 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:44:59 dist-test-slave-g8hp krb5kdc[15765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:44:59 dist-test-slave-g8hp krb5kdc[15765](info): set up 2 sockets
Sep 29 14:44:59 dist-test-slave-g8hp krb5kdc[15765](info): commencing operation
krb5kdc: starting...
W20260929 14:45:00.484923 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:45:00 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:45:00.609450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:00.723228 15778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:00.723548 15778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:00.723591 15778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:00.727937 15778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:45:00.728032 15778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:45:00.728048 15778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:45:00.728061 15778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:45:00.728080 15778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:00.728096 15778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:00.728111 15778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:45:00.728127 15778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:45:00.734020 15778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:00.735229 15778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:00.736346 15778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:00.747545 15783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:00.747612 15784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:00.747653 15778 server_base.cc:1034] running on GCE node
W20260929 14:45:00.747566 15786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:00.748577 15778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:00.749977 15778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:00.751199 15778 hybrid_clock.cc:648] HybridClock initialized: now 1790693100751206 us; error 79 us; skew 500 ppm
Sep 29 14:45:00 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:00.754896 15778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:45:00.756036 15778 webserver.cc:460] Webserver started at http://127.0.98.62:33707/ using document root <none> and password file <none>
I20260929 14:45:00.756656 15778 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:00.756736 15778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:00.756984 15778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:45:00.759344 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "8f217a6a9b01463bb6a3b0a6ca83b889"
format_stamp: "Formatted at 2026-09-29 14:45:00 on dist-test-slave-g8hp"
server_key: "9aa15e33d98368cd11b1afd2f56acb84"
server_key_iv: "059add419e370f001d3e3cf82f831b01"
server_key_version: "encryptionkey@0"
I20260929 14:45:00.759971 15778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f217a6a9b01463bb6a3b0a6ca83b889"
format_stamp: "Formatted at 2026-09-29 14:45:00 on dist-test-slave-g8hp"
server_key: "9aa15e33d98368cd11b1afd2f56acb84"
server_key_iv: "059add419e370f001d3e3cf82f831b01"
server_key_version: "encryptionkey@0"
I20260929 14:45:00.764235 15778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:45:00.767436 15793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:00.769367 15778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:45:00.769558 15778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "8f217a6a9b01463bb6a3b0a6ca83b889"
format_stamp: "Formatted at 2026-09-29 14:45:00 on dist-test-slave-g8hp"
server_key: "9aa15e33d98368cd11b1afd2f56acb84"
server_key_iv: "059add419e370f001d3e3cf82f831b01"
server_key_version: "encryptionkey@0"
I20260929 14:45:00.769786 15778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:00.784351 15778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:00.789067 15778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:00.789309 15778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:00.797082 15778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35641
I20260929 14:45:00.797706 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35641 every 8 connection(s)
I20260929 14:45:00.799054 15778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:45:00.802834 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:00.806404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15778
I20260929 14:45:00.806573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:45:00.806883 392 external_mini_cluster.cc:1428] Setting key b08b7419f3a942e73b9b85f8df40e1ae
I20260929 14:45:00.810529 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap starting.
I20260929 14:45:00.813578 15846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:00.814441 15846 log.cc:824] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:00.817324 15846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: No bootstrap required, opened a new log
Sep 29 14:45:00 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:00.821897 15846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:00.822263 15846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:00.822356 15846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Initialized, Role: FOLLOWER
I20260929 14:45:00.822953 15846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:00.823153 15846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:45:00.823220 15846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:45:00.823331 15846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:00.824676 15846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:00.825127 15846 leader_election.cc:302] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f217a6a9b01463bb6a3b0a6ca83b889; no voters:
I20260929 14:45:00.825491 15846 leader_election.cc:288] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:45:00.825762 15851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:45:00.826221 15851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 LEADER]: Becoming Leader. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Running, Role: LEADER
I20260929 14:45:00.826766 15851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:00.826897 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:00.808726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.98.62:35641)
0929 14:45:00.809278 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:00.809297 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:00.809341 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:45:00.810078 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:00.810100 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:00.810519 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:00.810922 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:00.811949 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:00.813664 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:00.814815 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:00.815693 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:00.815870 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:00.819241 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:00.819251 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:00.819277 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:00.819342 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:00.821924 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:00.822822 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:00.822829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:00.822836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:00.823030 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:00.823533 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:00.823537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:00.823538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:00.824147 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:00.824363 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:00.824715 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 14:45:00.825115 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":188,"threads_started":1}
I20260929 14:45:00.827855 15846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:45:00.829037 15853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f217a6a9b01463bb6a3b0a6ca83b889. Latest consensus state: current_term: 1 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:00.829216 15853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
I20260929 14:45:00.829595 15852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:00.829687 15852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
W20260929 14:45:00.834295 15866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:45:00.834411 15866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:45:00.834519 15860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:45:00.835462 15860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:45:00.844815 15860 catalog_manager.cc:1347] Generated new cluster ID: 52c11fde3c1d4ead93ad5dbb5a62d80a
I20260929 14:45:00.844941 15860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:45:00.861756 15860 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:45:00.863631 15860 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:45:00.881455 15860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Generated new TSK 0
I20260929 14:45:00.882825 15860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:45:00.931597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:01.063028 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:01.063349 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:01.063410 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:01.067533 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:01.067607 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:01.067687 15872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:45:01.073180 15872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:01.074393 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:01.075508 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:01.085187 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:01.085184 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:01.085320 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:01.085613 15872 server_base.cc:1034] running on GCE node
I20260929 14:45:01.086122 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:01.086756 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:01.087924 15872 hybrid_clock.cc:648] HybridClock initialized: now 1790693101087858 us; error 76 us; skew 500 ppm
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:01.090700 15872 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:45:01.091573 15872 webserver.cc:460] Webserver started at http://127.0.98.1:39111/ using document root <none> and password file <none>
I20260929 14:45:01.092119 15872 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:01.092180 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:01.092347 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:45:01.094475 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "edccd7ac0b9249b38c65d6ef75004c49"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "73925135ff459db6309d95ab4fb5047e"
server_key_iv: "883ce9e7772ecab583bf8f2853156fd4"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.095017 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "edccd7ac0b9249b38c65d6ef75004c49"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "73925135ff459db6309d95ab4fb5047e"
server_key_iv: "883ce9e7772ecab583bf8f2853156fd4"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.098987 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:45:01.101764 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.103261 15872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:45:01.103420 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "edccd7ac0b9249b38c65d6ef75004c49"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "73925135ff459db6309d95ab4fb5047e"
server_key_iv: "883ce9e7772ecab583bf8f2853156fd4"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.103544 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:01.121614 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:01.124920 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:01.125119 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:01.125720 15872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:01.126579 15872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:01.126631 15872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.126683 15872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:01.126710 15872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.136065 15872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41111
I20260929 14:45:01.136168 16000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41111 every 8 connection(s)
I20260929 14:45:01.137104 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:45:01.139377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 15872
I20260929 14:45:01.139525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:45:01.139858 392 external_mini_cluster.cc:1428] Setting key 59b87b1fd56fb79c1ab7bf81659f2e54
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:45:01.151860 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.139196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51953 (local address 127.0.98.62:35641)
0929 14:45:01.139391 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.139396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.140502 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 14:45:01.141432 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.141435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.141484 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.141570 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:01.142948 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.143505 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.144684 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.145055 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.145170 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.147678 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:01.147682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:01.147685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:01.147715 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:01.149510 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.150174 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.150177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.150178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.150262 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.150597 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.150600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.150602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.150842 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:01.150952 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.151559 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.151693 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 14:45:01.153693 16003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.139561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:51953)
0929 14:45:01.140338 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.140395 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.141229 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.141705 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.141712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.142066 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:01.142735 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.142796 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.143636 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.143639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.144550 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.144560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.145237 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.145241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.145409 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.146020 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:01.146038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:01.147483 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:01.149640 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.149647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.149663 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.150012 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.150354 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.150356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.150358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.150515 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.150973 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:01.150988 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:01.151293 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.151471 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.151848 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":118,"threads_started":1}
I20260929 14:45:01.155125 16001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:01.155411 16001 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:01.156067 16001 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:01.157886 15803 ts_manager.cc:194] Registered new tserver with Master: edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:01.159080 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:45:01.168848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:01.283399 16006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:01.283672 16006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:01.283730 16006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:01.287971 16006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:01.288049 16006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:01.288113 16006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:45:01.293655 16006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:01.294926 16006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:01.296000 16006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:01.306320 16014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:01.306380 16012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:01.306433 16006 server_base.cc:1034] running on GCE node
W20260929 14:45:01.306388 16011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:01.307044 16006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:01.307799 16006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:01.308971 16006 hybrid_clock.cc:648] HybridClock initialized: now 1790693101308926 us; error 59 us; skew 500 ppm
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:01.312353 16006 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:45:01.313791 16006 webserver.cc:460] Webserver started at http://127.0.98.2:46669/ using document root <none> and password file <none>
I20260929 14:45:01.314460 16006 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:01.314541 16006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:01.314735 16006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:45:01.317099 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "8eaa50c1a7ca46babdaaf673325c7795"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "37405b2a75b07cdf2c0f7b0c04c1daf1"
server_key_iv: "c6268c3c2c93370cc23e3c65c40cc0eb"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.317664 16006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8eaa50c1a7ca46babdaaf673325c7795"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "37405b2a75b07cdf2c0f7b0c04c1daf1"
server_key_iv: "c6268c3c2c93370cc23e3c65c40cc0eb"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.321668 16006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:45:01.324373 16021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.325793 16006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:45:01.325901 16006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "8eaa50c1a7ca46babdaaf673325c7795"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "37405b2a75b07cdf2c0f7b0c04c1daf1"
server_key_iv: "c6268c3c2c93370cc23e3c65c40cc0eb"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.326016 16006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:01.341109 16006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:01.344278 16006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:01.344453 16006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:01.345098 16006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:01.345992 16006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:01.346046 16006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.346093 16006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:01.346122 16006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.354867 16006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34253
I20260929 14:45:01.354907 16134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34253 every 8 connection(s)
I20260929 14:45:01.355837 16006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:45:01.356104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16006
I20260929 14:45:01.356206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:45:01.356508 392 external_mini_cluster.cc:1428] Setting key 1d6a71005f9a56f5062551262eebf0db
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:01.374523 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.357756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60393 (local address 127.0.98.62:35641)
0929 14:45:01.357889 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.357893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.359141 (+ 1248us) server_negotiation.cc:407] Connection header received
0929 14:45:01.360220 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.360223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.360282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.360377 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:01.361979 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.362599 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.364275 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.364509 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.364595 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.370447 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:01.370452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:01.370455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:01.370489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:01.372080 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.372742 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.372745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.372746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.372834 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.373255 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.373257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.373258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.373451 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:01.373527 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.374257 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.374393 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:45:01.375805 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.358235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:60393)
0929 14:45:01.358976 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.359035 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.359980 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.360562 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.360569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.361051 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:01.361776 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.361795 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.362724 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.362728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.364135 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.364146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.364669 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.364673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.364909 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.365579 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:01.365603 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:01.370205 (+ 4602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:01.372239 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.372246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.372266 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.372634 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.372946 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.372949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.372951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.373157 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.373548 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:01.373555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:01.373963 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.374208 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.374625 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":114,"threads_started":1}
I20260929 14:45:01.377439 16135 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:01.377729 16135 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:01.378397 16135 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:01.379733 15803 ts_manager.cc:194] Registered new tserver with Master: 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
I20260929 14:45:01.380519 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:45:01.388639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:01.502737 16140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:01.502995 16140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:01.503037 16140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:01.507290 16140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:01.507375 16140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:01.507444 16140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:45:01.513267 16140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:01.514504 16140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:01.515550 16140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:45:01.527768 16140 server_base.cc:1034] running on GCE node
W20260929 14:45:01.527725 16145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:01.527915 16148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:01.527729 16146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:01.528914 16140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:01.529820 16140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:01.531009 16140 hybrid_clock.cc:648] HybridClock initialized: now 1790693101531003 us; error 107 us; skew 500 ppm
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:01.534175 16140 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:45:01.535264 16140 webserver.cc:460] Webserver started at http://127.0.98.3:46743/ using document root <none> and password file <none>
I20260929 14:45:01.535827 16140 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:01.535897 16140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:01.536095 16140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:45:01.538306 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f64c4afd4d549ce9819d28bd825220f"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "6d5c987ad3dd56a2a12e937d319098cb"
server_key_iv: "84d3a4b52db2485f03fa3af777241287"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.538899 16140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f64c4afd4d549ce9819d28bd825220f"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "6d5c987ad3dd56a2a12e937d319098cb"
server_key_iv: "84d3a4b52db2485f03fa3af777241287"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.542938 16140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:45:01.545724 16155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.547241 16140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:45:01.547384 16140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0f64c4afd4d549ce9819d28bd825220f"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "6d5c987ad3dd56a2a12e937d319098cb"
server_key_iv: "84d3a4b52db2485f03fa3af777241287"
server_key_version: "encryptionkey@0"
I20260929 14:45:01.547510 16140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:01.569717 16140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:01.573021 16140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:01.573200 16140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:01.573748 16140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:01.574626 16140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:01.574678 16140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.574720 16140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:01.574739 16140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:01.583015 16140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38715
I20260929 14:45:01.583083 16268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38715 every 8 connection(s)
I20260929 14:45:01.583958 16140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:45:01.585606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16140
I20260929 14:45:01.585745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:45:01.586004 392 external_mini_cluster.cc:1428] Setting key 4776b250f9f77c888b04b9571bbab2e1
Sep 29 14:45:01 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:01.599097 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.586185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54269 (local address 127.0.98.62:35641)
0929 14:45:01.586355 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.586359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.587380 (+ 1021us) server_negotiation.cc:407] Connection header received
0929 14:45:01.588924 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.588927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.588989 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.589061 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:01.590322 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.590857 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.592080 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.592308 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.592396 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.595274 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:01.595278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:01.595280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:01.595307 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:01.596703 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.597430 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.597433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.597435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.597521 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.597905 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.597908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.597909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.598105 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:01.598187 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.598793 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.598925 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 14:45:01.599980 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.586680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:54269)
0929 14:45:01.587222 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.587283 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.588707 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.589249 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.589257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.589637 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:01.590181 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.590196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.590995 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.590999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.591946 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.591957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.592462 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.592466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.592648 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.593356 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:01.593375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:01.595057 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:01.596829 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.596835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.596852 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.597280 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.597635 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:01.597637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:01.597639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:01.597807 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:01.598188 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:01.598193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:01.598524 (+ 331us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.598708 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.599073 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260929 14:45:01.600970 16269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:01.601205 16269 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:01.601693 16269 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:01.602638 15803 ts_manager.cc:194] Registered new tserver with Master: 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715)
I20260929 14:45:01.603178 15803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54269
I20260929 14:45:01.612186 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:45:01.621327 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.614218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.98.62:35641)
0929 14:45:01.614333 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.614336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.614458 (+ 122us) server_negotiation.cc:407] Connection header received
0929 14:45:01.614564 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.614577 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.614615 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.614723 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:01.615436 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.615979 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.616849 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.617107 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.617219 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.618587 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:01.618590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:01.618592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:01.618637 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:01.619852 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.620207 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.620210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.620212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.620287 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:01.620499 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:01.620502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:01.620503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:01.620734 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:01.620833 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.621102 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.621204 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:45:01.624382 15803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:45:01.627619 15803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:45:01.643222 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.637534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.0.1:60886)
0929 14:45:01.637947 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.637960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.638045 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.638884 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.638887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.638906 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:01.639000 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.639007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640133 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.640136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.641593 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.641604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642664 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.642668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.642781 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.642822 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.642911 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.643039 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":53,"thread_start_us":119,"threads_started":1}
I20260929 14:45:01.643361 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.636825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.0.1:54518)
0929 14:45:01.637342 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.637407 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.637544 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.638285 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.638291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.638316 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:01.638469 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.638477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640505 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.640508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.641517 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.641528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642808 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.642812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.642974 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.643009 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.643065 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.643120 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":125,"thread_start_us":101,"threads_started":1}
I20260929 14:45:01.643481 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.637055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.0.98.2:34253)
0929 14:45:01.637392 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.637404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.637491 (+ 87us) server_negotiation.cc:407] Connection header received
0929 14:45:01.637749 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.637755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.637972 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.638640 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:01.639151 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640369 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.641657 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642482 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.642593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.642753 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.643233 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.643340 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260929 14:45:01.643530 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.637813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.0.98.3:38715)
0929 14:45:01.638279 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.638289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.638693 (+ 404us) server_negotiation.cc:407] Connection header received
0929 14:45:01.638778 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.638782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.638964 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.639106 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:01.639254 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640342 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.641811 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642486 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.642582 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.642753 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.643296 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.643399 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":107,"threads_started":1}
I20260929 14:45:01.643744 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.637632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.0.1:39220)
0929 14:45:01.638469 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.638482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.638564 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.639075 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.639079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.639096 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:01.639180 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.639187 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640481 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.640484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.641706 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.641714 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642812 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.642815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.642969 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.643018 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.643084 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.643153 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":98,"thread_start_us":107,"threads_started":1}
I20260929 14:45:01.644495 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.637972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.98.1:41111)
0929 14:45:01.638258 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.638271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.638296 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:45:01.638370 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.638374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.638724 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.638901 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:01.639274 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.640002 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.641748 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.642530 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.644022 (+ 1492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.644198 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.644247 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.644342 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260929 14:45:01.646752 16203 tablet_service.cc:1467] Processing CreateTablet for tablet e18abf1993ef415682276219394e9b0e (DEFAULT_TABLE table=test-table [id=634ccc2cdb794475bdd4534d5f479634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:01.647234 15935 tablet_service.cc:1467] Processing CreateTablet for tablet e18abf1993ef415682276219394e9b0e (DEFAULT_TABLE table=test-table [id=634ccc2cdb794475bdd4534d5f479634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:01.648011 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18abf1993ef415682276219394e9b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:01.648309 15935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18abf1993ef415682276219394e9b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:01.654897 16069 tablet_service.cc:1467] Processing CreateTablet for tablet e18abf1993ef415682276219394e9b0e (DEFAULT_TABLE table=test-table [id=634ccc2cdb794475bdd4534d5f479634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:01.656044 16069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18abf1993ef415682276219394e9b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:01.656353 16286 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap starting.
I20260929 14:45:01.657301 16287 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Bootstrap starting.
I20260929 14:45:01.658769 16286 tablet_bootstrap.cc:654] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:01.659677 16286 log.cc:824] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:01.659896 16287 tablet_bootstrap.cc:654] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:01.660652 16287 log.cc:824] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:01.661991 16286 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: No bootstrap required, opened a new log
I20260929 14:45:01.662199 16286 ts_tablet_manager.cc:1397] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 14:45:01.662524 16290 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Bootstrap starting.
I20260929 14:45:01.662659 16287 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: No bootstrap required, opened a new log
I20260929 14:45:01.662837 16287 ts_tablet_manager.cc:1397] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:45:01.664830 16290 tablet_bootstrap.cc:654] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:01.665562 16290 log.cc:824] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:01.666810 16287 raft_consensus.cc:348] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.666810 16286 raft_consensus.cc:348] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.667102 16287 raft_consensus.cc:374] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:01.667102 16286 raft_consensus.cc:374] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:01.667169 16286 raft_consensus.cc:729] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edccd7ac0b9249b38c65d6ef75004c49, State: Initialized, Role: FOLLOWER
I20260929 14:45:01.667200 16287 raft_consensus.cc:729] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f64c4afd4d549ce9819d28bd825220f, State: Initialized, Role: FOLLOWER
I20260929 14:45:01.667323 16290 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: No bootstrap required, opened a new log
I20260929 14:45:01.667514 16290 ts_tablet_manager.cc:1397] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:45:01.667725 16286 consensus_queue.cc:260] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.667725 16287 consensus_queue.cc:260] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.668800 16287 ts_tablet_manager.cc:1428] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:45:01.668800 16286 ts_tablet_manager.cc:1428] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:45:01.669159 16001 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
I20260929 14:45:01.669185 16269 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
I20260929 14:45:01.671744 16290 raft_consensus.cc:348] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.672067 16290 raft_consensus.cc:374] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:01.672178 16290 raft_consensus.cc:729] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eaa50c1a7ca46babdaaf673325c7795, State: Initialized, Role: FOLLOWER
I20260929 14:45:01.672689 16290 consensus_queue.cc:260] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.673449 16135 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
I20260929 14:45:01.673796 16290 ts_tablet_manager.cc:1428] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:45:01.757218 16292 raft_consensus.cc:482] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:01.757443 16292 raft_consensus.cc:504] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.758566 16292 leader_election.cc:288] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253), edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:01.762748 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.758982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49649 (local address 127.0.98.1:41111)
0929 14:45:01.759150 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.759154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.759171 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:45:01.759239 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.759242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.759290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.759361 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:01.759723 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.760419 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.761624 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.762131 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.762214 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.762276 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.762514 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.762578 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:45:01.762785 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.758871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.98.3:49649)
0929 14:45:01.759010 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.759025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.759125 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.759382 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.759386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.759410 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:01.759540 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.759548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.760593 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.760597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.761482 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.761492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.762257 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.762260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.762384 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.762411 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.762491 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.762571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 14:45:01.763046 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.758872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.98.3:58409)
0929 14:45:01.759377 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.759390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:01.759513 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:01.759788 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:01.759791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:01.759809 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:01.759916 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.759922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.760827 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.760829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.761798 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:01.761809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.762639 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.762643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:01.762743 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.762768 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:45:01.762841 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 14:45:01.762908 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":163,"thread_start_us":120,"threads_started":1}
I20260929 14:45:01.763424 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.759019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58409 (local address 127.0.98.2:34253)
0929 14:45:01.759159 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.759164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.759454 (+ 290us) server_negotiation.cc:407] Connection header received
0929 14:45:01.759617 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.759621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.759689 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.759759 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:01.760038 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.760709 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.761924 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.762525 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.762620 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.762679 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.763250 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.763329 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:45:01.763559 15955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18abf1993ef415682276219394e9b0e" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edccd7ac0b9249b38c65d6ef75004c49" is_pre_election: true
I20260929 14:45:01.763895 15955 raft_consensus.cc:2454] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 0.
I20260929 14:45:01.763988 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18abf1993ef415682276219394e9b0e" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa50c1a7ca46babdaaf673325c7795" is_pre_election: true
I20260929 14:45:01.764271 16089 raft_consensus.cc:2454] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 0.
I20260929 14:45:01.764386 16158 leader_election.cc:302] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f64c4afd4d549ce9819d28bd825220f, edccd7ac0b9249b38c65d6ef75004c49; no voters:
I20260929 14:45:01.764664 16292 raft_consensus.cc:2781] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:45:01.764756 16292 raft_consensus.cc:482] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:45:01.764788 16292 raft_consensus.cc:3037] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:01.765852 16292 raft_consensus.cc:504] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.766191 16292 leader_election.cc:288] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 election: Requested vote from peers 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253), edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:01.766530 15955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18abf1993ef415682276219394e9b0e" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edccd7ac0b9249b38c65d6ef75004c49"
I20260929 14:45:01.766645 15955 raft_consensus.cc:3037] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:01.766690 16089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18abf1993ef415682276219394e9b0e" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa50c1a7ca46babdaaf673325c7795"
I20260929 14:45:01.766815 16089 raft_consensus.cc:3037] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:01.767692 15955 raft_consensus.cc:2454] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 1.
I20260929 14:45:01.767849 16089 raft_consensus.cc:2454] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 1.
I20260929 14:45:01.767968 16158 leader_election.cc:302] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f64c4afd4d549ce9819d28bd825220f, edccd7ac0b9249b38c65d6ef75004c49; no voters:
I20260929 14:45:01.768136 16292 raft_consensus.cc:2781] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:45:01.768378 16292 raft_consensus.cc:686] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 1 LEADER]: Becoming Leader. State: Replica: 0f64c4afd4d549ce9819d28bd825220f, State: Running, Role: LEADER
I20260929 14:45:01.768687 16292 consensus_queue.cc:237] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:01.772078 15810 catalog_manager.cc:5495] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f64c4afd4d549ce9819d28bd825220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } health_report { overall_health: UNKNOWN } } }
I20260929 14:45:01.799307 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.795328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.98.3:38715)
0929 14:45:01.795467 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:01.795471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:01.795484 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:45:01.795582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:01.795585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:01.795641 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:01.795736 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:45:01.795944 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.796507 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.797714 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:01.797962 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:01.798159 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:01.798460 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:45:01.798886 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:45:01.798984 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:01.799079 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 14:45:01.799141 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 14:45:01.806034 16089 raft_consensus.cc:1261] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Refusing update from remote peer 0f64c4afd4d549ce9819d28bd825220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:45:01.806075 15955 raft_consensus.cc:1261] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Refusing update from remote peer 0f64c4afd4d549ce9819d28bd825220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:45:01.806809 16296 consensus_queue.cc:1035] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:45:01.807016 16292 consensus_queue.cc:1035] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:45:01.816515 16300 mvcc.cc:205] Tried to move back new op lower bound from 7334678944989179904 to 7334678944847306752. Current Snapshot: MvccSnapshot[applied={T|T < 7334678944989179904}]
I20260929 14:45:01.817348 16301 mvcc.cc:205] Tried to move back new op lower bound from 7334678944989179904 to 7334678944847306752. Current Snapshot: MvccSnapshot[applied={T|T < 7334678944989179904}]
I20260929 14:45:01.825258 15805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:45:01.825428 15805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:45:01.828087 15805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Sending DeleteTablet for 3 replicas of tablet e18abf1993ef415682276219394e9b0e
I20260929 14:45:01.829169 16069 tablet_service.cc:1514] Processing DeleteTablet for tablet e18abf1993ef415682276219394e9b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54518
I20260929 14:45:01.829206 15935 tablet_service.cc:1514] Processing DeleteTablet for tablet e18abf1993ef415682276219394e9b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60886
I20260929 14:45:01.829316 16203 tablet_service.cc:1514] Processing DeleteTablet for tablet e18abf1993ef415682276219394e9b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39220
I20260929 14:45:01.829818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15872
I20260929 14:45:01.829830 16309 tablet_replica.cc:330] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: stopping tablet replica
I20260929 14:45:01.830166 16309 raft_consensus.cc:2229] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:45:01.830523 16309 raft_consensus.cc:2258] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:01.830633 16308 tablet_replica.cc:330] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: stopping tablet replica
I20260929 14:45:01.831179 16308 raft_consensus.cc:2229] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:45:01.831580 16308 raft_consensus.cc:2258] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:01.832450 16309 ts_tablet_manager.cc:1905] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:01.833177 16308 ts_tablet_manager.cc:1905] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:01.835392 16309 ts_tablet_manager.cc:1918] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:45:01.835506 16309 log.cc:1196] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/e18abf1993ef415682276219394e9b0e
I20260929 14:45:01.835863 16309 ts_tablet_manager.cc:1939] T e18abf1993ef415682276219394e9b0e P 0f64c4afd4d549ce9819d28bd825220f: Deleting consensus metadata
I20260929 14:45:01.836333 16308 ts_tablet_manager.cc:1918] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:45:01.836496 16308 log.cc:1196] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/e18abf1993ef415682276219394e9b0e
I20260929 14:45:01.836833 16308 ts_tablet_manager.cc:1939] T e18abf1993ef415682276219394e9b0e P 8eaa50c1a7ca46babdaaf673325c7795: Deleting consensus metadata
I20260929 14:45:01.837325 15797 catalog_manager.cc:4841] TS 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): tablet e18abf1993ef415682276219394e9b0e (table test-table [id=634ccc2cdb794475bdd4534d5f479634]) successfully deleted
W20260929 14:45:01.838112 15796 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41111 (error 108)
I20260929 14:45:01.838338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16006
I20260929 14:45:01.838323 15797 catalog_manager.cc:4841] TS 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): tablet e18abf1993ef415682276219394e9b0e (table test-table [id=634ccc2cdb794475bdd4534d5f479634]) successfully deleted
I20260929 14:45:01.844005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16140
I20260929 14:45:01.845376 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:01.844942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.0.1:60888)
0929 14:45:01.845200 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:01.845279 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260929 14:45:01.845611 15796 catalog_manager.cc:4584] TS edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e18abf1993ef415682276219394e9b0e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41111: connect: Connection refused (error 111)
I20260929 14:45:01.849417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 15778
I20260929 14:45:21.855228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--webserver_interface=127.0.98.62
--webserver_port=33707
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:21.995653 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:21.996097 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:21.996147 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:22.000903 16312 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:45:22.001019 16312 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:45:22.001036 16312 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:45:22.001050 16312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:45:22.001070 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:22.001096 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:22.001125 16312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:45:22.001147 16312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:45:22.007768 16312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:22.009409 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:22.010878 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:22.028290 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.028290 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:22.028380 16312 server_base.cc:1034] running on GCE node
W20260929 14:45:22.028306 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:22.029510 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:22.031402 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:22.032655 16312 hybrid_clock.cc:648] HybridClock initialized: now 1790693122032621 us; error 59 us; skew 500 ppm
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:22.042773 16312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:45:22.044378 16312 webserver.cc:460] Webserver started at http://127.0.98.62:33707/ using document root <none> and password file <none>
I20260929 14:45:22.045285 16312 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:22.045393 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:22.051129 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:45:22.054597 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.056716 16312 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:45:22.056994 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "8f217a6a9b01463bb6a3b0a6ca83b889"
format_stamp: "Formatted at 2026-09-29 14:45:00 on dist-test-slave-g8hp"
server_key: "9aa15e33d98368cd11b1afd2f56acb84"
server_key_iv: "059add419e370f001d3e3cf82f831b01"
server_key_version: "encryptionkey@0"
I20260929 14:45:22.057873 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:22.127831 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:22.132278 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:22.132508 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:22.141656 16312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35641
I20260929 14:45:22.141989 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35641 every 8 connection(s)
I20260929 14:45:22.143198 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:45:22.146574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16312
I20260929 14:45:22.147192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41111
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=39111
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:45:22.150367 16381 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:45:22.153149 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap starting.
I20260929 14:45:22.175589 16381 log.cc:824] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:22.186769 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:45:22.187274 16381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap complete.
I20260929 14:45:22.192574 16381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:22.193295 16381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Initialized, Role: FOLLOWER
I20260929 14:45:22.194046 16381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:22.194216 16381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:45:22.194303 16381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:45:22.194424 16381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:45:22.197263 16381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:22.197728 16381 leader_election.cc:302] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f217a6a9b01463bb6a3b0a6ca83b889; no voters:
I20260929 14:45:22.198160 16381 leader_election.cc:288] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 14:45:22.198262 16385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:45:22.198532 16385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 LEADER]: Becoming Leader. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Running, Role: LEADER
I20260929 14:45:22.199011 16385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:22.199894 16381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:45:22.200716 16386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:22.200892 16386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
I20260929 14:45:22.201243 16387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f217a6a9b01463bb6a3b0a6ca83b889. Latest consensus state: current_term: 2 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:22.201326 16387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
I20260929 14:45:22.201745 16392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:45:22.208289 16392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:45:22.209219 16392 catalog_manager.cc:1259] Loaded cluster ID: 52c11fde3c1d4ead93ad5dbb5a62d80a
I20260929 14:45:22.209298 16392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:45:22.212126 16392 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:45:22.213140 16392 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Loaded TSK: 0
I20260929 14:45:22.240243 16392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Generated new TSK 1
I20260929 14:45:22.240671 16392 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:45:22.292912 16383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:22.293462 16383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:22.293571 16383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:22.298451 16383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:22.298556 16383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:22.298652 16383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:45:22.305306 16383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39111
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:22.306803 16383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:22.308210 16383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:22.321497 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.321497 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.321497 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:22.322288 16383 server_base.cc:1034] running on GCE node
I20260929 14:45:22.322932 16383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:22.324090 16383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:22.325309 16383 hybrid_clock.cc:648] HybridClock initialized: now 1790693122325254 us; error 70 us; skew 500 ppm
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:22.329890 16383 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:45:22.331703 16383 webserver.cc:460] Webserver started at http://127.0.98.1:39111/ using document root <none> and password file <none>
I20260929 14:45:22.332789 16383 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:22.333009 16383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:22.340595 16383 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 14:45:22.344812 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.347195 16383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:45:22.347416 16383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "edccd7ac0b9249b38c65d6ef75004c49"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "73925135ff459db6309d95ab4fb5047e"
server_key_iv: "883ce9e7772ecab583bf8f2853156fd4"
server_key_version: "encryptionkey@0"
I20260929 14:45:22.348455 16383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:22.369721 16383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:22.374836 16383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:22.375105 16383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:22.376055 16383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:22.377750 16425 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:45:22.383342 16383 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:45:22.383479 16383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260929 14:45:22.383561 16383 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:45:22.386214 16383 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:45:22.386297 16383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260929 14:45:22.386687 16425 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap starting.
I20260929 14:45:22.400665 16383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41111
I20260929 14:45:22.400731 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41111 every 8 connection(s)
I20260929 14:45:22.402257 16383 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:45:22.407514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16383
I20260929 14:45:22.408113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34253
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=46669
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:45:22.410833 16425 log.cc:824] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Log is configured to *not* fsync() on all Append() calls
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:22.421962 16425 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:45:22.422503 16425 tablet_bootstrap.cc:492] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap complete.
I20260929 14:45:22.423782 16425 ts_tablet_manager.cc:1397] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.012s
I20260929 14:45:22.428042 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.405459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:34831)
0929 14:45:22.406268 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.406340 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.407525 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.409694 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.409708 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.410358 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:22.411201 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.411231 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.413267 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.413272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.415038 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.415059 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.416302 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.416307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.416548 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.417600 (+ 1052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:22.417628 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:22.419989 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:22.423446 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.423455 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.423475 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.423941 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.424527 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.424530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.424533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.424735 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.425671 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:22.425679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:22.426089 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.426326 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.426800 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":1,"thread_start_us":177,"threads_started":1}
I20260929 14:45:22.428043 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.405444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34831 (local address 127.0.98.62:35641)
0929 14:45:22.406048 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.406079 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.406529 (+ 450us) server_negotiation.cc:407] Connection header received
0929 14:45:22.408285 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.408308 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.408998 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.409546 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.411466 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.413022 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.415254 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.416106 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.416317 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.420403 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.420414 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.420440 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.420498 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.423220 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.424145 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.424155 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.424163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.424401 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.424844 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.424848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.424849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.425527 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.425684 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.426501 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.426890 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":213,"threads_started":1}
I20260929 14:45:22.427502 16425 raft_consensus.cc:348] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.428750 16425 raft_consensus.cc:729] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edccd7ac0b9249b38c65d6ef75004c49, State: Initialized, Role: FOLLOWER
I20260929 14:45:22.429538 16425 consensus_queue.cc:260] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.431015 16425 ts_tablet_manager.cc:1428] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:45:22.431763 16533 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:22.432113 16533 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:22.433012 16533 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:22.437218 16345 ts_manager.cc:194] Registered new tserver with Master: edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:22.442143 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34831
I20260929 14:45:22.446507 16533 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 14:45:22.456146 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.441622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.0.1:54896)
0929 14:45:22.442541 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.442607 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.442805 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.443654 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.443662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.444280 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:22.444441 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.444449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.445606 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.445610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.446623 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.446638 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.447401 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.447424 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.447618 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.448413 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:22.448429 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:22.450388 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:22.453040 (+ 2652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.453062 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.453070 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.453564 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.454195 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.454200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.454202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.454358 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.455217 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:22.455227 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:22.455406 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.455562 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.455901 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":155,"threads_started":1}
I20260929 14:45:22.456146 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.441792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.98.1:41111)
0929 14:45:22.442735 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.442749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.442785 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:45:22.443016 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.443023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.443296 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.443476 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.444594 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.445466 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.446812 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.447289 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.447400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.450614 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.450629 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.450644 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.450683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.452766 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.453781 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.453789 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.453791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.454029 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.454500 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.454504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.454506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.455044 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.455252 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.455701 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.455933 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":178,"threads_started":1}
I20260929 14:45:22.457677 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet e18abf1993ef415682276219394e9b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260929 14:45:22.459061 16542 tablet_replica.cc:330] stopping tablet replica
I20260929 14:45:22.459538 16542 raft_consensus.cc:2229] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:45:22.459990 16542 raft_consensus.cc:2258] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:22.461558 16542 ts_tablet_manager.cc:1905] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:22.468261 16542 ts_tablet_manager.cc:1918] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:45:22.468432 16542 log.cc:1196] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/e18abf1993ef415682276219394e9b0e
I20260929 14:45:22.468727 16542 ts_tablet_manager.cc:1939] T e18abf1993ef415682276219394e9b0e P edccd7ac0b9249b38c65d6ef75004c49: Deleting consensus metadata
I20260929 14:45:22.471846 16331 catalog_manager.cc:4841] TS edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111): tablet e18abf1993ef415682276219394e9b0e (table test-table [id=634ccc2cdb794475bdd4534d5f479634]) successfully deleted
W20260929 14:45:22.566711 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:22.567202 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:22.567287 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:22.572158 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:22.572290 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:22.572371 16537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:45:22.579212 16537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46669
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:22.580830 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:22.582294 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:22.595194 16548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.595194 16547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.595194 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:22.595670 16537 server_base.cc:1034] running on GCE node
I20260929 14:45:22.596225 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:22.597153 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:22.598346 16537 hybrid_clock.cc:648] HybridClock initialized: now 1790693122598323 us; error 45 us; skew 500 ppm
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:22.602159 16537 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:45:22.603370 16537 webserver.cc:460] Webserver started at http://127.0.98.2:46669/ using document root <none> and password file <none>
I20260929 14:45:22.604079 16537 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:22.604161 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:22.609483 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:45:22.612218 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.613951 16537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:45:22.614168 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "8eaa50c1a7ca46babdaaf673325c7795"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "37405b2a75b07cdf2c0f7b0c04c1daf1"
server_key_iv: "c6268c3c2c93370cc23e3c65c40cc0eb"
server_key_version: "encryptionkey@0"
I20260929 14:45:22.614908 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:22.642719 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:22.648123 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:22.648346 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:22.649312 16537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:22.650517 16537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:22.650578 16537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.650632 16537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:22.650655 16537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.661764 16537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34253
I20260929 14:45:22.661783 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34253 every 8 connection(s)
I20260929 14:45:22.663301 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:45:22.668608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16537
I20260929 14:45:22.669262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38715
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=46743
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:22.681368 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.665930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60887 (local address 127.0.98.62:35641)
0929 14:45:22.666130 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.666135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.667770 (+ 1635us) server_negotiation.cc:407] Connection header received
0929 14:45:22.668939 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.668944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.669093 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.669208 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.671095 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.671753 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.673294 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.673571 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.673678 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.676828 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.676832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.676834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.676902 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.678542 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.679234 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.679238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.679240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.679339 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.679784 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.679789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.679790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.680047 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.680149 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.680941 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.681158 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:45:22.682621 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.666601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:60887)
0929 14:45:22.667555 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.667614 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.668633 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.669468 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.669481 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.670082 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:22.670838 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.670863 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.671933 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.671936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.673134 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.673150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.673775 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.673780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.674003 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.674749 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:22.674771 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:22.676558 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:22.678713 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.678723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.678748 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.679110 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.679439 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.679442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.679445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.679672 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.680168 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:22.680178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:22.680604 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.680831 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.681419 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":179,"threads_started":1}
I20260929 14:45:22.684345 16671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:22.684677 16671 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:22.685451 16671 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:22.687107 16345 ts_manager.cc:194] Registered new tserver with Master: 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
I20260929 14:45:22.688027 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60887
W20260929 14:45:22.806668 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:22.806968 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:22.807013 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:22.811200 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:22.811275 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:22.811337 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:45:22.816948 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46743
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:22.818171 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:22.819553 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:22.830442 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.830457 16680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:22.830469 16682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:22.830664 16674 server_base.cc:1034] running on GCE node
I20260929 14:45:22.831331 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:22.832056 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:22.833237 16674 hybrid_clock.cc:648] HybridClock initialized: now 1790693122833214 us; error 38 us; skew 500 ppm
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:22.836733 16674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:45:22.837952 16674 webserver.cc:460] Webserver started at http://127.0.98.3:46743/ using document root <none> and password file <none>
I20260929 14:45:22.838634 16674 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:22.838727 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:22.843582 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:45:22.846289 16689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.847756 16674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:45:22.847929 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0f64c4afd4d549ce9819d28bd825220f"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "6d5c987ad3dd56a2a12e937d319098cb"
server_key_iv: "84d3a4b52db2485f03fa3af777241287"
server_key_version: "encryptionkey@0"
I20260929 14:45:22.848526 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:22.867522 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:22.871245 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:22.871454 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:22.872134 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:22.873111 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:22.873172 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.873230 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:22.873256 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:22.882583 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38715
I20260929 14:45:22.882594 16802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38715 every 8 connection(s)
I20260929 14:45:22.883689 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:45:22.887974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16674
Sep 29 14:45:22 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:45:22.897272 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.888738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.0.98.62:35641)
0929 14:45:22.889034 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.889038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.889157 (+ 119us) server_negotiation.cc:407] Connection header received
0929 14:45:22.889293 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.889301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.889348 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.889454 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.890341 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.890958 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.891890 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.892185 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.892268 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.893840 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.893843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.893845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.893870 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.895428 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.895872 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.895876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.895878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.895958 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.896208 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.896211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.896213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.896450 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.896735 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.896994 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.897103 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":119,"threads_started":1}
I20260929 14:45:22.899148 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.885823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35221 (local address 127.0.98.62:35641)
0929 14:45:22.886018 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.886023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.887277 (+ 1254us) server_negotiation.cc:407] Connection header received
0929 14:45:22.888295 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.888316 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.888411 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.888533 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.890201 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.890791 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.892088 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.892366 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.892464 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.895130 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.895134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.895136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.895174 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.896607 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.897330 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.897333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.897335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.897442 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.897927 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.897930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.897932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.898113 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.898208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.898861 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.898987 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 14:45:22.900113 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.886250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:35221)
0929 14:45:22.887094 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.887138 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.887999 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.888795 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.888803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.889361 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:22.890024 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.890040 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.890935 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.890939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.891968 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.891978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.892516 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.892519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.892677 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.893192 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:22.893208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:22.894944 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:22.896721 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.896727 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.896747 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.897206 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.897541 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:22.897544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:22.897546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:22.897810 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:22.898231 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:22.898239 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:22.898579 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.898753 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.899044 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":125,"threads_started":1}
I20260929 14:45:22.901448 16803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35641
I20260929 14:45:22.901871 16803 heartbeater.cc:464] Registering TS with master...
I20260929 14:45:22.902479 16803 heartbeater.cc:511] Master 127.0.98.62:35641 requested a full tablet report, sending...
I20260929 14:45:22.903721 16345 ts_manager.cc:194] Registered new tserver with Master: 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715)
I20260929 14:45:22.904364 16345 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35221
I20260929 14:45:22.911393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:45:22.915824 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.912097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.98.62:35641)
0929 14:45:22.912266 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.912270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.912399 (+ 129us) server_negotiation.cc:407] Connection header received
0929 14:45:22.912528 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.912530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.912680 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.912779 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:45:22.913081 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.913631 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.914618 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.914855 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.914945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.915294 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:45:22.915450 (+ 156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 14:45:22.915697 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60144: token expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 14:45:22.915916 16535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60144: token expired
W20260929 14:45:22.916318 16273 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:35641)
I20260929 14:45:22.924093 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.916794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.0.98.62:35641)
0929 14:45:22.916946 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.916949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.916960 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:45:22.916991 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.916993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.917038 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.917112 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:22.917725 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.918229 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.919120 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.919377 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.919467 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.920927 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:22.920930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:22.920933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:22.920958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:22.922439 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.922936 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.922939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.922941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.923022 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:22.923240 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:22.923243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:22.923244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:22.923436 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:22.923540 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.923868 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.923969 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:45:22.926858 16345 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:45:22.930419 16345 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:45:22.938872 16467 tablet_service.cc:1467] Processing CreateTablet for tablet a327502a31d94ee5bb46814743efe220 (DEFAULT_TABLE table=test-table [id=9252c33523bf43378530888d8b8f108d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:22.939636 16467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327502a31d94ee5bb46814743efe220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:22.942741 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.937619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.0.1:34892)
0929 14:45:22.937798 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.937812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.937888 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.938830 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.938833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.938851 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:22.938960 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.938965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.940250 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.940265 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.941454 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.941466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.942290 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.942293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.942387 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.942453 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.942521 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.942603 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 14:45:22.942930 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.937828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.0.1:55258)
0929 14:45:22.938303 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:22.938315 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:22.938400 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:22.939178 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:22.939180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:22.939197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:22.939283 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.939288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.940681 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.940684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.941494 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:22.941504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.942571 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.942575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:22.942661 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.942684 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:45:22.942732 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 14:45:22.942801 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":117,"threads_started":1}
I20260929 14:45:22.943495 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.937795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.98.3:38715)
0929 14:45:22.938306 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.938322 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.938346 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:45:22.938430 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.938436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.938672 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.938852 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:22.939054 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.940113 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.941587 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.942166 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.942264 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.942423 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.943273 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.943380 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":132,"threads_started":1}
I20260929 14:45:22.943493 16815 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap starting.
I20260929 14:45:22.943559 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:22.938311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.0.98.2:34253)
0929 14:45:22.938691 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:22.938703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:22.938726 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:45:22.938815 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:22.938820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:22.939025 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:22.939396 (+ 371us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:22.939541 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.940574 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.941620 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:22.942426 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:22.943092 (+ 666us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:22.943247 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:22.943316 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 14:45:22.943410 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":170,"threads_started":1}
I20260929 14:45:22.944574 16815 tablet_bootstrap.cc:654] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:22.945886 16815 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: No bootstrap required, opened a new log
I20260929 14:45:22.945979 16815 ts_tablet_manager.cc:1397] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 14:45:22.946422 16815 raft_consensus.cc:348] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.946558 16815 raft_consensus.cc:374] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:22.946594 16815 raft_consensus.cc:729] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edccd7ac0b9249b38c65d6ef75004c49, State: Initialized, Role: FOLLOWER
I20260929 14:45:22.946753 16815 consensus_queue.cc:260] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.947000 16737 tablet_service.cc:1467] Processing CreateTablet for tablet a327502a31d94ee5bb46814743efe220 (DEFAULT_TABLE table=test-table [id=9252c33523bf43378530888d8b8f108d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:22.947256 16605 tablet_service.cc:1467] Processing CreateTablet for tablet a327502a31d94ee5bb46814743efe220 (DEFAULT_TABLE table=test-table [id=9252c33523bf43378530888d8b8f108d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:45:22.947700 16815 ts_tablet_manager.cc:1428] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 14:45:22.948189 16737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327502a31d94ee5bb46814743efe220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:22.948227 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a327502a31d94ee5bb46814743efe220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:45:22.956111 16818 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Bootstrap starting.
I20260929 14:45:22.956650 16819 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Bootstrap starting.
I20260929 14:45:22.958848 16819 tablet_bootstrap.cc:654] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:22.959286 16818 tablet_bootstrap.cc:654] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Neither blocks nor log segments found. Creating new log.
I20260929 14:45:22.960209 16819 log.cc:824] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:22.960218 16818 log.cc:824] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:22.962673 16819 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: No bootstrap required, opened a new log
I20260929 14:45:22.962673 16818 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: No bootstrap required, opened a new log
I20260929 14:45:22.962918 16819 ts_tablet_manager.cc:1397] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:45:22.962918 16818 ts_tablet_manager.cc:1397] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:45:22.967164 16819 raft_consensus.cc:348] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.967164 16818 raft_consensus.cc:348] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.967521 16819 raft_consensus.cc:374] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:22.967521 16818 raft_consensus.cc:374] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:45:22.967577 16819 raft_consensus.cc:729] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f64c4afd4d549ce9819d28bd825220f, State: Initialized, Role: FOLLOWER
I20260929 14:45:22.967577 16818 raft_consensus.cc:729] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eaa50c1a7ca46babdaaf673325c7795, State: Initialized, Role: FOLLOWER
I20260929 14:45:22.968135 16819 consensus_queue.cc:260] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.968135 16818 consensus_queue.cc:260] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:22.968937 16803 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
I20260929 14:45:22.969231 16671 heartbeater.cc:503] Master 127.0.98.62:35641 was elected leader, sending a full tablet report...
I20260929 14:45:22.969344 16819 ts_tablet_manager.cc:1428] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 14:45:22.970414 16818 ts_tablet_manager.cc:1428] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:45:23.122610 16822 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:23.122839 16822 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:23.124068 16822 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253), edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:23.127606 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.124340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.98.3:44083)
0929 14:45:23.124481 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:23.124497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:23.124598 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:23.124887 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:23.124890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:23.124903 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:23.125026 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:23.125033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.125759 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.125762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:23.126545 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:23.126554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.127184 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.127187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:23.127296 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:23.127313 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:45:23.127376 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:45:23.127450 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 14:45:23.127606 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.124479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44083 (local address 127.0.98.2:34253)
0929 14:45:23.124607 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:23.124611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:23.124628 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:45:23.124689 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:23.124693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:23.124748 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:23.124843 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:23.125127 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.125657 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.126672 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.127077 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.127166 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:23.127234 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:23.127407 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 14:45:23.127481 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:45:23.128228 16824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.124358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.98.3:48693)
0929 14:45:23.124786 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:23.124796 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:23.124891 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:23.125225 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:23.125228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:23.125246 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:45:23.125355 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:23.125361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.126082 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.126086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:23.126870 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:23.126880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.127851 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.127854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:23.127943 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:23.127966 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:45:23.128026 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 14:45:23.128089 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":109,"thread_start_us":114,"threads_started":1}
I20260929 14:45:23.128266 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327502a31d94ee5bb46814743efe220" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa50c1a7ca46babdaaf673325c7795" is_pre_election: true
I20260929 14:45:23.128634 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.124554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48693 (local address 127.0.98.1:41111)
0929 14:45:23.124975 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:23.124980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:23.124992 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:45:23.125040 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:23.125042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:23.125109 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:23.125213 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:45:23.125469 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.125981 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.126998 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.127464 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.127562 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:23.127757 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:23.128401 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 14:45:23.128487 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":184,"threads_started":1}
I20260929 14:45:23.128635 16625 raft_consensus.cc:2454] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 0.
I20260929 14:45:23.129133 16693 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795; no voters:
I20260929 14:45:23.129133 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327502a31d94ee5bb46814743efe220" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edccd7ac0b9249b38c65d6ef75004c49" is_pre_election: true
I20260929 14:45:23.129410 16487 raft_consensus.cc:2454] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 0.
I20260929 14:45:23.129462 16822 raft_consensus.cc:2781] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:45:23.129567 16822 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:45:23.129631 16822 raft_consensus.cc:3037] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:23.130669 16822 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:23.131000 16822 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 election: Requested vote from peers 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253), edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111)
I20260929 14:45:23.131357 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327502a31d94ee5bb46814743efe220" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eaa50c1a7ca46babdaaf673325c7795"
I20260929 14:45:23.131381 16487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a327502a31d94ee5bb46814743efe220" candidate_uuid: "0f64c4afd4d549ce9819d28bd825220f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edccd7ac0b9249b38c65d6ef75004c49"
I20260929 14:45:23.131492 16487 raft_consensus.cc:3037] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:23.131484 16625 raft_consensus.cc:3037] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:45:23.132495 16487 raft_consensus.cc:2454] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 1.
I20260929 14:45:23.132546 16625 raft_consensus.cc:2454] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f64c4afd4d549ce9819d28bd825220f in term 1.
I20260929 14:45:23.132808 16692 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f64c4afd4d549ce9819d28bd825220f, edccd7ac0b9249b38c65d6ef75004c49; no voters:
I20260929 14:45:23.133010 16822 raft_consensus.cc:2781] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:45:23.133235 16822 raft_consensus.cc:686] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 1 LEADER]: Becoming Leader. State: Replica: 0f64c4afd4d549ce9819d28bd825220f, State: Running, Role: LEADER
I20260929 14:45:23.133596 16822 consensus_queue.cc:237] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
W20260929 14:45:23.134687 16804 tablet.cc:2369] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:45:23.137408 16345 catalog_manager.cc:5495] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f64c4afd4d549ce9819d28bd825220f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } health_report { overall_health: UNKNOWN } } }
W20260929 14:45:23.154029 16534 tablet.cc:2369] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:45:23.164700 16672 tablet.cc:2369] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:45:23.183962 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.180407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.0.98.3:38715)
0929 14:45:23.180536 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:23.180540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:23.180552 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:45:23.180625 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:23.180627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:23.180680 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:23.180773 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:45:23.181082 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.181588 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.182371 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:23.182578 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:23.182652 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:23.182979 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:45:23.183436 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:45:23.183536 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:23.183595 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 14:45:23.183650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:45:23.190558 16625 raft_consensus.cc:1261] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Refusing update from remote peer 0f64c4afd4d549ce9819d28bd825220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:45:23.190555 16487 raft_consensus.cc:1261] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Refusing update from remote peer 0f64c4afd4d549ce9819d28bd825220f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:45:23.191300 16826 consensus_queue.cc:1035] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:45:23.191581 16822 consensus_queue.cc:1035] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:45:23.200671 16830 mvcc.cc:205] Tried to move back new op lower bound from 7334679032580755456 to 7334679032358187008. Current Snapshot: MvccSnapshot[applied={T|T < 7334679032580755456}]
I20260929 14:45:23.208581 16342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:45:23.208832 16342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:45:23.211354 16342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Sending DeleteTablet for 3 replicas of tablet a327502a31d94ee5bb46814743efe220
I20260929 14:45:23.212109 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet a327502a31d94ee5bb46814743efe220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54896
I20260929 14:45:23.212246 16737 tablet_service.cc:1514] Processing DeleteTablet for tablet a327502a31d94ee5bb46814743efe220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34892
I20260929 14:45:23.212396 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet a327502a31d94ee5bb46814743efe220 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:45:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55258
I20260929 14:45:23.212507 16838 tablet_replica.cc:330] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: stopping tablet replica
I20260929 14:45:23.212627 16838 raft_consensus.cc:2229] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:45:23.212837 16838 raft_consensus.cc:2258] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:23.212960 16839 tablet_replica.cc:330] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: stopping tablet replica
I20260929 14:45:23.213305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16383
I20260929 14:45:23.213367 16839 raft_consensus.cc:2229] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:45:23.214156 16839 raft_consensus.cc:2258] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:23.215657 16840 tablet_replica.cc:330] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: stopping tablet replica
I20260929 14:45:23.215873 16839 ts_tablet_manager.cc:1905] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:23.216102 16840 raft_consensus.cc:2229] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:45:23.216506 16840 raft_consensus.cc:2258] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:45:23.218441 16840 ts_tablet_manager.cc:1905] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:23.219100 16839 ts_tablet_manager.cc:1918] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:45:23.219206 16839 log.cc:1196] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/a327502a31d94ee5bb46814743efe220
I20260929 14:45:23.219525 16839 ts_tablet_manager.cc:1939] T a327502a31d94ee5bb46814743efe220 P 0f64c4afd4d549ce9819d28bd825220f: Deleting consensus metadata
W20260929 14:45:23.220460 16331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41111 (error 108)
I20260929 14:45:23.221288 16332 catalog_manager.cc:4841] TS 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): tablet a327502a31d94ee5bb46814743efe220 (table test-table [id=9252c33523bf43378530888d8b8f108d]) successfully deleted
I20260929 14:45:23.221379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16537
I20260929 14:45:23.223405 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.223067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41111 (local address 127.0.0.1:53728)
0929 14:45:23.223249 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:23.223318 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 14:45:23.223578 16331 catalog_manager.cc:4584] TS edccd7ac0b9249b38c65d6ef75004c49 (127.0.98.1:41111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a327502a31d94ee5bb46814743efe220: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41111: connect: Connection refused (error 111)
I20260929 14:45:23.227926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16674
I20260929 14:45:23.229072 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:23.228741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.0.1:55264)
0929 14:45:23.228940 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:23.228988 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 14:45:23.233525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16312
I20260929 14:45:43.239797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--webserver_interface=127.0.98.62
--webserver_port=33707
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:43.386910 16842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:43.387441 16842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:43.387534 16842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:43.392899 16842 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 14:45:43.393018 16842 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 14:45:43.393035 16842 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 14:45:43.393049 16842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:45:43.393070 16842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:43.393088 16842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:43.393103 16842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:45:43.393118 16842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:45:43.401638 16842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:43.403353 16842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:43.404949 16842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:45:43.418094 16842 server_base.cc:1034] running on GCE node
W20260929 14:45:43.418023 16850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:43.418023 16848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:43.418040 16847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:43.419200 16842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:43.421038 16842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:43.422480 16842 hybrid_clock.cc:648] HybridClock initialized: now 1790693143422381 us; error 93 us; skew 500 ppm
Sep 29 14:45:43 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:43.427451 16842 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:45:43.429052 16842 webserver.cc:460] Webserver started at http://127.0.98.62:33707/ using document root <none> and password file <none>
I20260929 14:45:43.429908 16842 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:43.429999 16842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:43.436240 16842 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 14:45:43.440166 16857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:43.442355 16842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:45:43.442586 16842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "8f217a6a9b01463bb6a3b0a6ca83b889"
format_stamp: "Formatted at 2026-09-29 14:45:00 on dist-test-slave-g8hp"
server_key: "9aa15e33d98368cd11b1afd2f56acb84"
server_key_iv: "059add419e370f001d3e3cf82f831b01"
server_key_version: "encryptionkey@0"
I20260929 14:45:43.443470 16842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:43.475757 16842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:43.480718 16842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:43.481034 16842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:43.491006 16842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35641
I20260929 14:45:43.491372 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35641 every 8 connection(s)
I20260929 14:45:43.492633 16842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:45:43.499543 16910 sys_catalog.cc:263] Verifying existing consensus state
I20260929 14:45:43.500607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16842
I20260929 14:45:43.501320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41111
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=39111
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:45:43.503177 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap starting.
I20260929 14:45:43.528784 16910 log.cc:824] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Log is configured to *not* fsync() on all Append() calls
I20260929 14:45:43.545670 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:45:43.546181 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: Bootstrap complete.
I20260929 14:45:43.551553 16910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:43.552277 16910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Initialized, Role: FOLLOWER
I20260929 14:45:43.553131 16910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:43.553395 16910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:45:43.553501 16910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:45:43.553639 16910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:45:43.557320 16910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:43.557883 16910 leader_election.cc:302] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f217a6a9b01463bb6a3b0a6ca83b889; no voters:
I20260929 14:45:43.558388 16910 leader_election.cc:288] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 14:45:43.558573 16914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:45:43.558868 16914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [term 3 LEADER]: Becoming Leader. State: Replica: 8f217a6a9b01463bb6a3b0a6ca83b889, State: Running, Role: LEADER
I20260929 14:45:43.559273 16914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } }
I20260929 14:45:43.559971 16910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:45:43.560613 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f217a6a9b01463bb6a3b0a6ca83b889. Latest consensus state: current_term: 3 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:43.561396 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
I20260929 14:45:43.561273 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f217a6a9b01463bb6a3b0a6ca83b889" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35641 } } }
I20260929 14:45:43.562286 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889 [sys.catalog]: This master's current role is: LEADER
I20260929 14:45:43.566659 16930 catalog_manager.cc:1259] Loaded cluster ID: 52c11fde3c1d4ead93ad5dbb5a62d80a
I20260929 14:45:43.566776 16930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: loading cluster ID for follower catalog manager: success
I20260929 14:45:43.570077 16930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f217a6a9b01463bb6a3b0a6ca83b889: acquiring CA information for follower catalog manager: success
F20260929 14:45:43.571863 16930 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb3415c19ad google::LogMessage::Fail() at ??:0
@ 0x7fb3415c488a google::LogMessage::SendToLog() at ??:0
@ 0x7fb3415c14de google::LogMessage::Flush() at ??:0
@ 0x7fb3415c2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb34428518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb34bf26989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fb34bf03023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fb34bf0354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fb34befc652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fb34befb9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fb34bf42541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb343804512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb344e486db start_thread at ??:0
@ 0x7fb3400bf61f clone at ??:0
W20260929 14:45:43.694157 16912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:43.694589 16912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:43.694667 16912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:43.699998 16912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:43.700127 16912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:43.700230 16912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:45:43.707686 16912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39111
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:43.709440 16912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:43.710918 16912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:43.723942 16950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:43.724323 16951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:43.724323 16953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:43.724965 16912 server_base.cc:1034] running on GCE node
I20260929 14:45:43.725695 16912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:43.726856 16912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:43.728127 16912 hybrid_clock.cc:648] HybridClock initialized: now 1790693143728086 us; error 62 us; skew 500 ppm
Sep 29 14:45:43 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:43.732689 16912 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:45:43.734431 16912 webserver.cc:460] Webserver started at http://127.0.98.1:39111/ using document root <none> and password file <none>
I20260929 14:45:43.735448 16912 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:43.735674 16912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:43.743016 16912 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:45:43.747123 16960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:43.749572 16912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:45:43.749825 16912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "edccd7ac0b9249b38c65d6ef75004c49"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "73925135ff459db6309d95ab4fb5047e"
server_key_iv: "883ce9e7772ecab583bf8f2853156fd4"
server_key_version: "encryptionkey@0"
I20260929 14:45:43.750949 16912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:43.774255 16912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:43.779629 16912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:43.779897 16912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:43.780800 16912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:43.782711 16967 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 14:45:43.790999 16912 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 14:45:43.791149 16912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260929 14:45:43.791251 16912 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 14:45:43.794181 16912 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 14:45:43.794277 16912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 14:45:43.794740 16967 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap starting.
I20260929 14:45:43.806684 16912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41111
I20260929 14:45:43.806970 17074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41111 every 8 connection(s)
I20260929 14:45:43.808209 16912 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:45:43.812747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 16912
I20260929 14:45:43.813418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34253
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=46669
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:43.838925 17077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:45:43.810989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:54297)
0929 14:45:43.811691 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:43.811762 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:43.813174 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:43.837315 (+ 24141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":202,"threads_started":1}
W20260929 14:45:43.839821 17075 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35641 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35641: Client connection negotiation failed: client connection to 127.0.98.62:35641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 14:45:43.840982 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:43.840535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:54507)
0929 14:45:43.840716 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:43.840834 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 14:45:43.841892 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:43.841631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:39681)
0929 14:45:43.841757 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:43.841814 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 14:45:43.842159 17075 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:45:43.969442 16967 log_util.cc:230] Ignoring log segment corruption in /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/a327502a31d94ee5bb46814743efe220.recovery/wal-000000001 because there are no log entries following the corrupted one. The server probably crashed in the middle of writing an entry to the write-ahead log or downloaded an active log via tablet copy. Error detail: Corruption: CRC mismatch in log entry header: Log file corruption detected. Failed trying to read batch #0 at offset 577 for log segment /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-0/wal/wals/a327502a31d94ee5bb46814743efe220.recovery/wal-000000001: Prior entries: [off=506 REPLICATE (1.1)] [off=506 REPLICATE (1.2)] [off=537 COMMIT (1.1)] [off=577 COMMIT (1.2)]
I20260929 14:45:43.971617 16967 log.cc:824] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Log is configured to *not* fsync() on all Append() calls
W20260929 14:45:43.972846 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:43.973259 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:43.973326 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:43.979539 17078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:43.979681 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:43.979759 17078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:45:43.982852 16967 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:45:43.983419 16967 tablet_bootstrap.cc:492] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Bootstrap complete.
I20260929 14:45:43.985299 16967 ts_tablet_manager.cc:1397] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Time spent bootstrapping tablet: real 0.191s user 0.177s sys 0.012s
I20260929 14:45:43.990643 17078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46669
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:43.990226 16967 raft_consensus.cc:348] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:43.990962 16967 raft_consensus.cc:729] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edccd7ac0b9249b38c65d6ef75004c49, State: Initialized, Role: FOLLOWER
I20260929 14:45:43.991672 16967 consensus_queue.cc:260] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:43.992254 17078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:43.993193 16967 ts_tablet_manager.cc:1428] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 14:45:43.993508 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:43.993104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:51583)
0929 14:45:43.993299 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:43.993396 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 14:45:43.994037 17078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:45:43.994369 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:43.994095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:37581)
0929 14:45:43.994227 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:43.994289 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 14:45:44.009295 17086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:44.009295 17088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:44.009295 17085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:44.009711 17078 server_base.cc:1034] running on GCE node
I20260929 14:45:44.010516 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:44.011564 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:44.013060 17078 hybrid_clock.cc:648] HybridClock initialized: now 1790693144013009 us; error 79 us; skew 500 ppm
Sep 29 14:45:44 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:44.017508 17078 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:45:44.018996 17078 webserver.cc:460] Webserver started at http://127.0.98.2:46669/ using document root <none> and password file <none>
I20260929 14:45:44.019846 17078 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:44.019940 17078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:44.026201 17078 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:45:44.030308 17095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:44.032549 17078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:45:44.032807 17078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "8eaa50c1a7ca46babdaaf673325c7795"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "37405b2a75b07cdf2c0f7b0c04c1daf1"
server_key_iv: "c6268c3c2c93370cc23e3c65c40cc0eb"
server_key_version: "encryptionkey@0"
I20260929 14:45:44.033932 17078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:44.054452 17078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:44.059728 17078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:44.060027 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:44.061101 17078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:44.062887 17102 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 14:45:44.068464 17102 ts_tablet_manager.cc:1852] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 14:45:44.069151 17102 ts_tablet_manager.cc:1905] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:45:44.075281 17102 ts_tablet_manager.cc:1918] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:45:44.075486 17102 log.cc:1196] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/a327502a31d94ee5bb46814743efe220
I20260929 14:45:44.075883 17102 ts_tablet_manager.cc:1939] T a327502a31d94ee5bb46814743efe220 P 8eaa50c1a7ca46babdaaf673325c7795: Deleting consensus metadata
I20260929 14:45:44.076397 17078 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 14:45:44.076484 17078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260929 14:45:44.076567 17078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:44.076588 17078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:44.088243 17078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34253
I20260929 14:45:44.088311 17209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34253 every 8 connection(s)
I20260929 14:45:44.089578 17078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:45:44.094962 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.092916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:42835)
0929 14:45:44.093743 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.093925 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":186,"threads_started":1}
W20260929 14:45:44.095661 17210 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35641 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35641: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
I20260929 14:45:44.096534 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.096191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:43953)
0929 14:45:44.096316 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.096452 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 14:45:44.097518 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.097137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:44005)
0929 14:45:44.097293 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.097442 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 14:45:44.097712 17210 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:45:44.098637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17078
I20260929 14:45:44.099153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38715
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35641
--webserver_port=46743
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:45:44.245677 17213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:45:44.246124 17213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:45:44.246198 17213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:45:44.250895 17213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:45:44.251008 17213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:45:44.251077 17213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:45:44.257926 17213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46743
--tserver_master_addrs=127.0.98.62:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:45:44.259436 17213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:45:44.260773 17213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:45:44.274436 17219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:44.274585 17221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:45:44.274389 17218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:45:44.274757 17213 server_base.cc:1034] running on GCE node
I20260929 14:45:44.275386 17213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:45:44.276265 17213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:45:44.277462 17213 hybrid_clock.cc:648] HybridClock initialized: now 1790693144277476 us; error 78 us; skew 500 ppm
Sep 29 14:45:44 dist-test-slave-g8hp krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:45:44.281078 17213 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:45:44.282289 17213 webserver.cc:460] Webserver started at http://127.0.98.3:46743/ using document root <none> and password file <none>
I20260929 14:45:44.283015 17213 fs_manager.cc:359] Metadata directory not provided
I20260929 14:45:44.283092 17213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:45:44.288558 17213 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:45:44.291324 17228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:44.293131 17213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 14:45:44.293267 17213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0f64c4afd4d549ce9819d28bd825220f"
format_stamp: "Formatted at 2026-09-29 14:45:01 on dist-test-slave-g8hp"
server_key: "6d5c987ad3dd56a2a12e937d319098cb"
server_key_iv: "84d3a4b52db2485f03fa3af777241287"
server_key_version: "encryptionkey@0"
I20260929 14:45:44.293922 17213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:45:44.319624 17213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:45:44.323433 17213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:45:44.323639 17213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:45:44.324357 17213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:45:44.325361 17213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:45:44.325412 17213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:44.325467 17213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:45:44.325495 17213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:45:44.335320 17213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38715
I20260929 14:45:44.335338 17341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38715 every 8 connection(s)
I20260929 14:45:44.336493 17213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:45:44.339498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17213
I20260929 14:45:44.341280 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.339109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:33209)
0929 14:45:44.339726 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.340343 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":130,"threads_started":1}
W20260929 14:45:44.341773 17342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35641 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35641: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
I20260929 14:45:44.342389 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.342173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:60035)
0929 14:45:44.342286 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.342333 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 14:45:44.342975 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.342780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:40013)
0929 14:45:44.342883 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.342928 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:45:44.343279 17342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 14:45:44.995895 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:44.995363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:34409)
0929 14:45:44.995714 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:44.995783 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":97,"threads_started":1}
I20260929 14:45:45.098964 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.098475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:35763)
0929 14:45:45.098783 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:45.098855 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
I20260929 14:45:45.344389 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.343936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:59093)
0929 14:45:45.344201 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:45.344274 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260929 14:45:45.861492 17349 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:45.861763 17349 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:45.863006 17349 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715), 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
Sep 29 14:45:45 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 14:45:45 dist-test-slave-g8hp krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 14:45:45.882831 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.863675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34253 (local address 127.0.98.1:47219)
0929 14:45:45.864025 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:45.864035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:45.864101 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:45.865806 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:45.865812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:45.866377 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:45.866916 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:45.866933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.868478 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.868481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:45.869515 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:45.869525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.870256 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.870259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:45.870454 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:45.875540 (+ 5086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:45.875556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:45.877277 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:45.879865 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:45.879872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:45.879887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:45.880245 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:45.880733 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:45.880735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:45.880737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:45.880947 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:45.881755 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:45.881761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:45.882014 (+ 253us) client_negotiation.cc:769] Sending connection context
0929 14:45:45.882308 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 14:45:45.882639 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":146,"spinlock_wait_cycles":4352,"thread_start_us":73,"threads_started":1}
I20260929 14:45:45.883064 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.863921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47219 (local address 127.0.98.2:34253)
0929 14:45:45.864294 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:45.864303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:45.864335 (+ 32us) server_negotiation.cc:407] Connection header received
0929 14:45:45.865097 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:45.865104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:45.865430 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:45.865719 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:45.867109 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.868314 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.869646 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.870123 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.870284 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:45.877624 (+ 7340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:45.877633 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:45.877653 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:45.877696 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:45.879722 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:45.880402 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:45.880410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:45.880416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:45.880612 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:45.881038 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:45.881041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:45.881042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:45.881616 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:45.881762 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:45.882529 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 14:45:45.882885 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
I20260929 14:45:45.883078 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.863473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56893 (local address 127.0.98.3:38715)
0929 14:45:45.863806 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 14:45:45.863823 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:45:45.864010 (+ 187us) server_negotiation.cc:407] Connection header received
0929 14:45:45.864928 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:45:45.864938 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:45:45.865352 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:45:45.865728 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:45:45.867109 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.868313 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.869655 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.870122 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.870276 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:45.877826 (+ 7550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:45:45.877832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:45:45.877847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:45:45.877881 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:45:45.880301 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:45.880849 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:45.880855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:45.880887 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:45.881041 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:45:45.881435 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:45:45.881438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:45:45.881439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:45:45.881982 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:45:45.882147 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:45:45.882592 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 14:45:45.882890 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":84,"threads_started":1}
I20260929 14:45:45.882954 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.863280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38715 (local address 127.0.98.1:56893)
0929 14:45:45.863891 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:45.863907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:45:45.864027 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:45:45.865806 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:45:45.865812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:45:45.866378 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:45:45.866916 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:45.866933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.868471 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.868473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:45.869513 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:45:45.869524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:45:45.870253 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:45:45.870257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:45:45.870460 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:45:45.875454 (+ 4994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:45:45.875476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:45:45.877532 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:45:45.880429 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:45.880432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:45.880434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:45.880729 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:45.881129 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:45:45.881133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:45:45.881134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:45:45.881220 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:45:45.882220 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:45:45.882222 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:45:45.882320 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:45:45.882428 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 14:45:45.882742 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":266,"spinlock_wait_cycles":10880,"thread_start_us":66,"threads_started":1}
W20260929 14:45:45.886129 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
W20260929 14:45:45.886348 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
I20260929 14:45:45.886400 16964 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edccd7ac0b9249b38c65d6ef75004c49; no voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795
I20260929 14:45:45.886675 17349 raft_consensus.cc:2726] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:45:45.997361 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:45.997001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:44437)
0929 14:45:45.997186 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:45.997260 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 14:45:46.100455 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:46.099860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:38503)
0929 14:45:46.100263 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:46.100336 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":83,"threads_started":1}
I20260929 14:45:46.346179 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:46.345474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:54193)
0929 14:45:46.345890 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:46.345979 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":132,"threads_started":1}
I20260929 14:45:46.999185 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:46.998408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:45789)
0929 14:45:46.998925 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:46.999018 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":100,"threads_started":1}
I20260929 14:45:47.102334 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:47.101613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:47969)
0929 14:45:47.102043 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:47.102130 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260929 14:45:47.347924 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:47.347390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:38217)
0929 14:45:47.347726 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:47.347801 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
I20260929 14:45:47.887542 17359 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:47.887696 17359 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:47.888159 17359 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715), 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
W20260929 14:45:47.889403 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
W20260929 14:45:47.889537 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
I20260929 14:45:47.889599 16964 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edccd7ac0b9249b38c65d6ef75004c49; no voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795
I20260929 14:45:47.889823 17359 raft_consensus.cc:2726] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:45:48.000837 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:48.000284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:50443)
0929 14:45:48.000605 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:48.000685 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
I20260929 14:45:48.104282 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:48.103701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:51951)
0929 14:45:48.104091 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:48.104153 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":105,"threads_started":1}
I20260929 14:45:48.349366 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:48.348948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:39349)
0929 14:45:48.349206 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:48.349267 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260929 14:45:49.002486 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:49.001936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:48125)
0929 14:45:49.002303 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:49.002372 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":79,"threads_started":1}
I20260929 14:45:49.105543 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:49.105132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:45277)
0929 14:45:49.105387 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:49.105445 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260929 14:45:49.350757 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:49.350296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:38505)
0929 14:45:49.350596 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:49.350657 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260929 14:45:50.004040 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:50.003546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:46543)
0929 14:45:50.003869 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:50.003938 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260929 14:45:50.107009 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:50.106512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:57359)
0929 14:45:50.106831 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:50.106899 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":79,"threads_started":1}
I20260929 14:45:50.352238 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:50.351723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:38375)
0929 14:45:50.352045 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:50.352115 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20260929 14:45:50.970105 17369 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:50.970247 17369 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:50.970646 17369 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715), 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
W20260929 14:45:50.971511 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
W20260929 14:45:50.971649 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
I20260929 14:45:50.971707 16964 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edccd7ac0b9249b38c65d6ef75004c49; no voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795
I20260929 14:45:50.971939 17369 raft_consensus.cc:2726] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:45:51.005581 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:51.005096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:42905)
0929 14:45:51.005413 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:51.005480 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260929 14:45:51.108480 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:51.107927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:51875)
0929 14:45:51.108311 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:51.108380 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":83,"threads_started":1}
I20260929 14:45:51.353669 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:51.353199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:41055)
0929 14:45:51.353503 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:51.353559 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260929 14:45:52.007050 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:52.006523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:54167)
0929 14:45:52.006877 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:52.006946 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
I20260929 14:45:52.109692 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:52.109275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:47885)
0929 14:45:52.109539 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:52.109596 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260929 14:45:52.355022 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:52.354545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:44969)
0929 14:45:52.354860 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:52.354920 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":70,"threads_started":1}
I20260929 14:45:53.008598 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:53.008069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:35641)
0929 14:45:53.008421 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:53.008491 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
I20260929 14:45:53.100502 17377 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:53.100653 17377 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:53.101116 17377 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715), 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
W20260929 14:45:53.102048 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
W20260929 14:45:53.102170 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
I20260929 14:45:53.102470 16964 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edccd7ac0b9249b38c65d6ef75004c49; no voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795
I20260929 14:45:53.102682 17377 raft_consensus.cc:2726] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:45:53.111142 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:53.110665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:57329)
0929 14:45:53.110977 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:53.111043 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
I20260929 14:45:53.356585 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:53.356035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:34605)
0929 14:45:53.356388 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:53.356466 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":93,"threads_started":1}
I20260929 14:45:54.010542 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:54.009930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:56397)
0929 14:45:54.010297 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:54.010384 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260929 14:45:54.113039 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:54.112373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:40805)
0929 14:45:54.112779 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:54.112893 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":100,"threads_started":1}
I20260929 14:45:54.358347 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:54.357757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:36139)
0929 14:45:54.358138 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:54.358218 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260929 14:45:55.012107 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:55.011590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:55469)
0929 14:45:55.011930 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:55.011994 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260929 14:45:55.114459 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:55.113902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:52461)
0929 14:45:55.114186 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:55.114351 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
I20260929 14:45:55.359866 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:55.359383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:38767)
0929 14:45:55.359687 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:55.359755 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
I20260929 14:45:56.013622 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:56.013151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:44927)
0929 14:45:56.013448 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:56.013514 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260929 14:45:56.115918 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:56.115307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:49449)
0929 14:45:56.115699 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:56.115764 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260929 14:45:56.361235 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:56.360776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:60113)
0929 14:45:56.361068 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:56.361129 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260929 14:45:56.684624 17389 raft_consensus.cc:482] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:45:56.684772 17389 raft_consensus.cc:504] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f64c4afd4d549ce9819d28bd825220f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38715 } } peers { permanent_uuid: "8eaa50c1a7ca46babdaaf673325c7795" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34253 } } peers { permanent_uuid: "edccd7ac0b9249b38c65d6ef75004c49" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41111 } }
I20260929 14:45:56.685266 17389 leader_election.cc:288] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715), 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253)
W20260929 14:45:56.686349 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f64c4afd4d549ce9819d28bd825220f (127.0.98.3:38715): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
W20260929 14:45:56.686570 16964 leader_election.cc:341] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8eaa50c1a7ca46babdaaf673325c7795 (127.0.98.2:34253): Not found: Tablet not found: a327502a31d94ee5bb46814743efe220
I20260929 14:45:56.686636 16964 leader_election.cc:302] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edccd7ac0b9249b38c65d6ef75004c49; no voters: 0f64c4afd4d549ce9819d28bd825220f, 8eaa50c1a7ca46babdaaf673325c7795
I20260929 14:45:56.686811 17389 raft_consensus.cc:2726] T a327502a31d94ee5bb46814743efe220 P edccd7ac0b9249b38c65d6ef75004c49 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:45:57.015005 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:57.014558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:51425)
0929 14:45:57.014834 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:57.014897 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260929 14:45:57.117501 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:57.116979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:45155)
0929 14:45:57.117273 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:57.117383 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260929 14:45:57.362627 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:57.362136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:43721)
0929 14:45:57.362460 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:57.362526 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260929 14:45:58.016554 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:58.015989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:51575)
0929 14:45:58.016345 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:58.016426 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
I20260929 14:45:58.118847 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:58.118397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:37897)
0929 14:45:58.118672 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:58.118738 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260929 14:45:58.363750 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:58.363333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.3:46303)
0929 14:45:58.363591 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:58.363647 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260929 14:45:59.018193 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:59.017750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.1:45129)
0929 14:45:59.018026 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:59.018084 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260929 14:45:59.120239 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:45:59.119821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35641 (local address 127.0.98.2:54277)
0929 14:45:59.120073 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 14:45:59.120130 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":66,"threads_started":1}
I20260929 14:45:59.349200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 16912
I20260929 14:45:59.354749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17078
I20260929 14:45:59.359717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17213
2026-09-29T14:45:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59850 ms)
[----------] 30 tests from SecurityITest (243984 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:45:59 dist-test-slave-g8hp krb5kdc[17400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:45:59 dist-test-slave-g8hp krb5kdc[17400](info): set up 2 sockets
Sep 29 14:45:59 dist-test-slave-g8hp krb5kdc[17400](info): commencing operation
krb5kdc: starting...
W20260929 14:46:00.336776 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:00 dist-test-slave-g8hp krb5kdc[17400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:00.473021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:00.594247 17413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:00.594547 17413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:00.594605 17413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:00.598955 17413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:00.599036 17413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:00.599064 17413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:00.599084 17413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:00.599105 17413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:00.604753 17413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:00.605954 17413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:00.607136 17413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:00.616977 17418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:00.617106 17413 server_base.cc:1034] running on GCE node
W20260929 14:46:00.617116 17421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:00.616981 17419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:00.618116 17413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:00.619342 17413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:00.620540 17413 hybrid_clock.cc:648] HybridClock initialized: now 1790693160620506 us; error 50 us; skew 500 ppm
Sep 29 14:46:00 dist-test-slave-g8hp krb5kdc[17400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:00.634656 17413 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:00.635892 17413 webserver.cc:460] Webserver started at http://127.0.98.62:37355/ using document root <none> and password file <none>
I20260929 14:46:00.636489 17413 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:00.636566 17413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:00.636781 17413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:00.639075 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "81b30928bcd046699b1bfabb369db307"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "ad8ccaca9532e4799dfd1e908342c594"
server_key_iv: "c5ff26c1d94da80a21e9c245282927fa"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.639664 17413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "81b30928bcd046699b1bfabb369db307"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "ad8ccaca9532e4799dfd1e908342c594"
server_key_iv: "c5ff26c1d94da80a21e9c245282927fa"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.643927 17413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:00.646795 17428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:00.648329 17413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:00.648471 17413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "81b30928bcd046699b1bfabb369db307"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "ad8ccaca9532e4799dfd1e908342c594"
server_key_iv: "c5ff26c1d94da80a21e9c245282927fa"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.648623 17413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:00.668126 17413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:00.671746 17413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:00.671967 17413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:00.679479 17413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34583
I20260929 14:46:00.679512 17480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34583 every 8 connection(s)
I20260929 14:46:00.680542 17413 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:00.683924 17481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:00.690698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17413
I20260929 14:46:00.690850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:00.690888 17481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: Bootstrap starting.
I20260929 14:46:00.691145 392 external_mini_cluster.cc:1428] Setting key 87a6e0e0bf18ce53b7d734baa968efbe
I20260929 14:46:00.693907 17481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:00.694787 17481 log.cc:824] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:00.697063 17481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: No bootstrap required, opened a new log
Sep 29 14:46:00 dist-test-slave-g8hp krb5kdc[17400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:00.701447 17481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260929 14:46:00.701743 17481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:00.701821 17481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b30928bcd046699b1bfabb369db307, State: Initialized, Role: FOLLOWER
I20260929 14:46:00.702374 17481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260929 14:46:00.702538 17481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:00.702615 17481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:00.702728 17481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:00.703708 17481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260929 14:46:00.704069 17481 leader_election.cc:302] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b30928bcd046699b1bfabb369db307; no voters:
I20260929 14:46:00.704432 17481 leader_election.cc:288] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:00.704645 17486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:00.704892 17486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [term 1 LEADER]: Becoming Leader. State: Replica: 81b30928bcd046699b1bfabb369db307, State: Running, Role: LEADER
I20260929 14:46:00.705420 17486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } }
I20260929 14:46:00.705935 17481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:00.706843 17486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b30928bcd046699b1bfabb369db307. Latest consensus state: current_term: 1 leader_uuid: "81b30928bcd046699b1bfabb369db307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260929 14:46:00.707064 17486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:00.707175 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:00.692744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.0.98.62:34583)
0929 14:46:00.693397 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:00.693414 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:00.693453 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:46:00.694246 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:00.694269 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:00.694664 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:00.695057 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:00.695920 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.697015 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.697973 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.698653 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.698825 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:00.701497 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:00.701503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:00.701519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:00.701554 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:00.703523 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:00.704022 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:00.704029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:00.704036 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:00.704171 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:00.704538 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:00.704543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:00.704545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:00.705190 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:00.705397 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:00.705799 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 14:46:00.706169 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":150,"threads_started":1}
I20260929 14:46:00.707235 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b30928bcd046699b1bfabb369db307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b30928bcd046699b1bfabb369db307" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34583 } } }
I20260929 14:46:00.707531 17495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:00.707669 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:00.711035 17495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:00.717548 17495 catalog_manager.cc:1347] Generated new cluster ID: be664c7e455a4cd7910c77c489e1446e
I20260929 14:46:00.717626 17495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:00.729846 17495 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:00.730929 17495 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:00.740183 17495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: Generated new TSK 0
I20260929 14:46:00.741010 17495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:46:00.781708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:00.898344 17507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:00.898625 17507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:00.898674 17507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:00.902814 17507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:00.902889 17507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:00.902951 17507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:00.909129 17507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:00.910264 17507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:00.911334 17507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:00.921881 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:00.921942 17515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:00.921905 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:00.922082 17507 server_base.cc:1034] running on GCE node
I20260929 14:46:00.922931 17507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:00.923676 17507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:00.924901 17507 hybrid_clock.cc:648] HybridClock initialized: now 1790693160924816 us; error 98 us; skew 500 ppm
Sep 29 14:46:00 dist-test-slave-g8hp krb5kdc[17400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:00.927958 17507 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:46:00.928918 17507 webserver.cc:460] Webserver started at http://127.0.98.1:39605/ using document root <none> and password file <none>
I20260929 14:46:00.929522 17507 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:00.929595 17507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:00.929798 17507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:00.931934 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb5db78b40004e96a9050c5705c55569"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "2d4f0e19074e9cf1e124807a8348532b"
server_key_iv: "9c83665cda31c0086cb5a6de429b544b"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.932463 17507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb5db78b40004e96a9050c5705c55569"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "2d4f0e19074e9cf1e124807a8348532b"
server_key_iv: "9c83665cda31c0086cb5a6de429b544b"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.936396 17507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:00.939306 17522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:00.940641 17507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:00.940750 17507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "bb5db78b40004e96a9050c5705c55569"
format_stamp: "Formatted at 2026-09-29 14:46:00 on dist-test-slave-g8hp"
server_key: "2d4f0e19074e9cf1e124807a8348532b"
server_key_iv: "9c83665cda31c0086cb5a6de429b544b"
server_key_version: "encryptionkey@0"
I20260929 14:46:00.940848 17507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:00.957455 17507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:00.960777 17507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:00.960989 17507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:00.961587 17507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:00.962422 17507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:00.962468 17507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:00.962520 17507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:00.962545 17507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:00.971460 17507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44181
I20260929 14:46:00.971479 17635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44181 every 8 connection(s)
I20260929 14:46:00.972409 17507 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:00.978605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17507
I20260929 14:46:00.978739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:00.979035 392 external_mini_cluster.cc:1428] Setting key 076524332d64b6dbcb0eaa50a9627901
Sep 29 14:46:00 dist-test-slave-g8hp krb5kdc[17400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:00.987684 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:00.974277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34885 (local address 127.0.98.62:34583)
0929 14:46:00.974403 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:00.974407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:00.975263 (+ 856us) server_negotiation.cc:407] Connection header received
0929 14:46:00.976667 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:00.976671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:00.976728 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:00.976803 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:00.978034 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.978720 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.980037 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.980322 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.980445 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:00.983569 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:00.983572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:00.983575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:00.983602 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:00.985117 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:00.985847 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:00.985851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:00.985852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:00.985940 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:00.986375 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:00.986379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:00.986380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:00.986592 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:00.986672 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:00.987425 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 14:46:00.987550 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:46:00.988973 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:00.974574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.1:34885)
0929 14:46:00.975097 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:00.975144 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:00.976431 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:00.977019 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:00.977026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:00.977387 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:00.977875 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:00.977888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.978889 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.978892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:00.979915 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:00.979926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:00.980532 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:00.980537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:00.980761 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:00.981446 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:00.981469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:00.983367 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:00.985257 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:00.985265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:00.985286 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:00.985720 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:00.986047 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:00.986050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:00.986052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:00.986261 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:00.986716 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:00.986724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:00.987115 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 14:46:00.987355 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 14:46:00.987765 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:46:00.990520 17636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260929 14:46:00.990772 17636 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:00.991412 17636 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:46:00.993239 17445 ts_manager.cc:194] Registered new tserver with Master: bb5db78b40004e96a9050c5705c55569 (127.0.98.1:44181)
I20260929 14:46:00.994443 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:46:01.013675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:01.127048 17641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:01.127350 17641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:01.127396 17641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:01.131573 17641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:01.131652 17641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:01.131714 17641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:01.137348 17641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:01.138487 17641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:01.139546 17641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:01.149689 17647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:01.149727 17646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:01.149822 17641 server_base.cc:1034] running on GCE node
W20260929 14:46:01.149744 17649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:01.150331 17641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:01.151052 17641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:01.152212 17641 hybrid_clock.cc:648] HybridClock initialized: now 1790693161152189 us; error 35 us; skew 500 ppm
Sep 29 14:46:01 dist-test-slave-g8hp krb5kdc[17400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:01.155045 17641 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:46:01.156028 17641 webserver.cc:460] Webserver started at http://127.0.98.2:33771/ using document root <none> and password file <none>
I20260929 14:46:01.156584 17641 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:01.156659 17641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:01.156853 17641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:01.159035 17641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "804aecd67c74413da60f507c8c6f7bb5"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "ad7c2371458dbe35ccd1011946baab08"
server_key_iv: "476ba6cd3659665ac597cf84cc5caa39"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.159693 17641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "804aecd67c74413da60f507c8c6f7bb5"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "ad7c2371458dbe35ccd1011946baab08"
server_key_iv: "476ba6cd3659665ac597cf84cc5caa39"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.163580 17641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:01.166451 17656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.167821 17641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:46:01.167945 17641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "804aecd67c74413da60f507c8c6f7bb5"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "ad7c2371458dbe35ccd1011946baab08"
server_key_iv: "476ba6cd3659665ac597cf84cc5caa39"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.168064 17641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:01.185078 17641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:01.188349 17641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:01.188537 17641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:01.189183 17641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:01.190079 17641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:01.190145 17641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.190194 17641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:01.190217 17641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.199486 17641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37893
I20260929 14:46:01.199512 17769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37893 every 8 connection(s)
I20260929 14:46:01.200457 17641 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:46:01.210592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17641
I20260929 14:46:01.210697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
Sep 29 14:46:01 dist-test-slave-g8hp krb5kdc[17400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:01.211113 392 external_mini_cluster.cc:1428] Setting key 8756095b6fa7941fe6fb2b336c908122
I20260929 14:46:01.215345 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.202457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48103 (local address 127.0.98.62:34583)
0929 14:46:01.202635 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.202638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.203518 (+ 880us) server_negotiation.cc:407] Connection header received
0929 14:46:01.204992 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.204995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.205054 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.205132 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:01.206616 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.207141 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.208463 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.208675 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.208764 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.211433 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:01.211438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:01.211440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:01.211473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:01.213259 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.213742 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.213745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.213747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.213837 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.214200 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.214203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.214205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.214435 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:01.214512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.215042 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.215184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 14:46:01.216463 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.202835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.2:48103)
0929 14:46:01.203332 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.203381 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.204739 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.205332 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.205340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.205742 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:01.206392 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.206410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.207276 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.207279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.208332 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.208343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.208830 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.208834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.209037 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.209652 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:01.209670 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:01.211185 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:01.213354 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:01.213360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:01.213375 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:01.213644 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:01.213915 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:01.213918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:01.213919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:01.214111 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:01.214524 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:01.214529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:01.214789 (+ 260us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.214967 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.215385 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260929 14:46:01.217831 17770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260929 14:46:01.218138 17770 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:01.218683 17770 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
I20260929 14:46:01.219846 17445 ts_manager.cc:194] Registered new tserver with Master: 804aecd67c74413da60f507c8c6f7bb5 (127.0.98.2:37893)
I20260929 14:46:01.220510 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48103
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:46:01.241428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:01.357422 17775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:01.357715 17775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:01.357761 17775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:01.361907 17775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:01.361976 17775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:01.362037 17775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:01.367532 17775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:01.368624 17775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:01.369627 17775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:01.379525 17783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:01.379509 17781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:01.379570 17780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:01.379629 17775 server_base.cc:1034] running on GCE node
I20260929 14:46:01.380347 17775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:01.381065 17775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:01.382256 17775 hybrid_clock.cc:648] HybridClock initialized: now 1790693161382235 us; error 33 us; skew 500 ppm
Sep 29 14:46:01 dist-test-slave-g8hp krb5kdc[17400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:01.385512 17775 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:46:01.386544 17775 webserver.cc:460] Webserver started at http://127.0.98.3:33161/ using document root <none> and password file <none>
I20260929 14:46:01.387116 17775 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:01.387192 17775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:01.387404 17775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:01.389639 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "120fc5237adf4a95b74d5fd8d7cce60d"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "99b3c2a2c67e5a45592d8373a7577686"
server_key_iv: "700f8401e4f6db726ee576dc2cd33453"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.390264 17775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "120fc5237adf4a95b74d5fd8d7cce60d"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "99b3c2a2c67e5a45592d8373a7577686"
server_key_iv: "700f8401e4f6db726ee576dc2cd33453"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.394271 17775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:01.397235 17790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.398674 17775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:01.398802 17775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "120fc5237adf4a95b74d5fd8d7cce60d"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "99b3c2a2c67e5a45592d8373a7577686"
server_key_iv: "700f8401e4f6db726ee576dc2cd33453"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.398918 17775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:01.433188 17775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:01.436722 17775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:01.436973 17775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:01.437602 17775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:01.438498 17775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:01.438550 17775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.438607 17775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:01.438640 17775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.447319 17775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45109
I20260929 14:46:01.447363 17903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45109 every 8 connection(s)
I20260929 14:46:01.448227 17775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:01.448753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17775
I20260929 14:46:01.448904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:01.449252 392 external_mini_cluster.cc:1428] Setting key b399e888ec54706f7307a9598d7d5cac
Sep 29 14:46:01 dist-test-slave-g8hp krb5kdc[17400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:01.463225 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.450395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53909 (local address 127.0.98.62:34583)
0929 14:46:01.450508 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.450512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.451615 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 14:46:01.453239 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.453242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.453300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.453373 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:01.454626 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.455155 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.456328 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.456554 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.456651 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.459360 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:01.459363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:01.459365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:01.459391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:01.460931 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.461470 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.461474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.461475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.461555 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.461884 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.461887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.461889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.462216 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:01.462306 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.462918 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.463060 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 14:46:01.464164 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.450805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34583 (local address 127.0.98.3:53909)
0929 14:46:01.451327 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.451511 (+ 184us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.453008 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.453571 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.453579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.453957 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:01.454461 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.454474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.455283 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.455287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.456192 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.456203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.456715 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.456719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.456934 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.457580 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:01.457598 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:01.459167 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:01.461070 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:01.461076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:01.461094 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:01.461364 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:01.461643 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:01.461646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:01.461649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:01.461792 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:01.462319 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:01.462326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:01.462677 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.462861 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.463222 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260929 14:46:01.465392 17904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34583
I20260929 14:46:01.465620 17904 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:01.466079 17904 heartbeater.cc:511] Master 127.0.98.62:34583 requested a full tablet report, sending...
I20260929 14:46:01.467113 17445 ts_manager.cc:194] Registered new tserver with Master: 120fc5237adf4a95b74d5fd8d7cce60d (127.0.98.3:45109)
I20260929 14:46:01.467645 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53909
I20260929 14:46:01.475630 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:01.486135 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.477643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.0.98.62:34583)
0929 14:46:01.477815 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.477818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.477863 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:46:01.478060 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.478062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.478111 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.478181 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:01.478979 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.479576 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.480581 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.480841 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.481061 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.482500 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:01.482503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:01.482505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:01.482535 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:01.483994 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.484450 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.484454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.484455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.484549 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:01.485043 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:01.485049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:01.485050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:01.485321 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:01.485443 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.485715 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.485850 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 14:46:01.489918 17445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:01.493240 17445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:01.508597 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.502632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.0.1:58232)
0929 14:46:01.503610 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.503622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.503690 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.504163 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.504168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.504191 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:01.504341 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.504348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505437 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.505440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.506762 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.506772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.507988 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.507992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.508110 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.508161 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.508268 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.508417 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":27,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20260929 14:46:01.508773 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.502423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.98.3:45109)
0929 14:46:01.503456 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.503467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.503490 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:46:01.503574 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.503579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.503859 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.504009 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:01.504491 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505737 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.507481 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.508090 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.508181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.508317 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.508498 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.508612 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":168,"threads_started":1}
I20260929 14:46:01.509001 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.502393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37893 (local address 127.0.0.1:53064)
0929 14:46:01.503347 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.503359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.503439 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.504390 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.504393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.504410 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:01.504481 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.504486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505847 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.505851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.507316 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.507325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.508356 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.508359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.508442 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.508460 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.508499 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.508566 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260929 14:46:01.509155 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.502276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45109 (local address 127.0.0.1:57918)
0929 14:46:01.502980 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.503034 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.503177 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.504185 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.504188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.504206 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:01.504369 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.504376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505869 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.505872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.507269 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.507279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.508212 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.508216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.508338 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.508362 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.508430 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.508520 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":96,"thread_start_us":90,"threads_started":1}
I20260929 14:46:01.509292 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.503203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.98.1:44181)
0929 14:46:01.503635 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.503643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.503667 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:46:01.503827 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.503832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.504017 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.504146 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:01.504467 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505325 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.506893 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.507851 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.507961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.508116 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.509034 (+ 918us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.509153 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":92,"threads_started":1}
I20260929 14:46:01.510257 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.503215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.0.98.2:37893)
0929 14:46:01.503892 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.503902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.503922 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:46:01.503991 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.503996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.504254 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.504398 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:01.504562 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.505712 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.507453 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.508235 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.509264 (+ 1029us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.509996 (+ 732us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.510047 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.510133 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":142,"threads_started":1}
I20260929 14:46:01.512142 17570 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f (DEFAULT_TABLE table=test-table [id=ad50ab61810841c0ac26ec81bf7b931d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:01.512477 17837 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f (DEFAULT_TABLE table=test-table [id=ad50ab61810841c0ac26ec81bf7b931d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:01.513227 17570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba7d196a07247408dbddd39b21f9f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:01.512846 17704 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f (DEFAULT_TABLE table=test-table [id=ad50ab61810841c0ac26ec81bf7b931d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:01.513553 17837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba7d196a07247408dbddd39b21f9f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:01.513816 17704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba7d196a07247408dbddd39b21f9f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:01.519668 17921 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: Bootstrap starting.
I20260929 14:46:01.520517 17922 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Bootstrap starting.
I20260929 14:46:01.521306 17923 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Bootstrap starting.
I20260929 14:46:01.523211 17922 tablet_bootstrap.cc:654] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:01.523211 17921 tablet_bootstrap.cc:654] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:01.524159 17921 log.cc:824] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:01.524389 17922 log.cc:824] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:01.525952 17923 tablet_bootstrap.cc:654] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:01.526520 17921 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: No bootstrap required, opened a new log
I20260929 14:46:01.526755 17921 ts_tablet_manager.cc:1397] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:46:01.526789 17923 log.cc:824] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:01.529680 17923 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: No bootstrap required, opened a new log
I20260929 14:46:01.529986 17923 ts_tablet_manager.cc:1397] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 14:46:01.530506 17922 tablet_bootstrap.cc:492] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: No bootstrap required, opened a new log
I20260929 14:46:01.530798 17922 ts_tablet_manager.cc:1397] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 14:46:01.531452 17921 raft_consensus.cc:348] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.531823 17921 raft_consensus.cc:374] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:01.531905 17921 raft_consensus.cc:729] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5db78b40004e96a9050c5705c55569, State: Initialized, Role: FOLLOWER
I20260929 14:46:01.532413 17921 consensus_queue.cc:260] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.533324 17636 heartbeater.cc:503] Master 127.0.98.62:34583 was elected leader, sending a full tablet report...
I20260929 14:46:01.533524 17921 ts_tablet_manager.cc:1428] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260929 14:46:01.533898 17923 raft_consensus.cc:348] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.534119 17923 raft_consensus.cc:374] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:01.534179 17923 raft_consensus.cc:729] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804aecd67c74413da60f507c8c6f7bb5, State: Initialized, Role: FOLLOWER
I20260929 14:46:01.534610 17923 consensus_queue.cc:260] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.535285 17770 heartbeater.cc:503] Master 127.0.98.62:34583 was elected leader, sending a full tablet report...
I20260929 14:46:01.534816 17922 raft_consensus.cc:348] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.535421 17922 raft_consensus.cc:374] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:01.535470 17922 raft_consensus.cc:729] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 120fc5237adf4a95b74d5fd8d7cce60d, State: Initialized, Role: FOLLOWER
I20260929 14:46:01.535585 17923 ts_tablet_manager.cc:1428] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:46:01.535856 17922 consensus_queue.cc:260] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.536572 17904 heartbeater.cc:503] Master 127.0.98.62:34583 was elected leader, sending a full tablet report...
I20260929 14:46:01.536778 17922 ts_tablet_manager.cc:1428] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:46:01.544878 17929 raft_consensus.cc:482] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:01.545096 17929 raft_consensus.cc:504] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.546394 17929 leader_election.cc:288] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804aecd67c74413da60f507c8c6f7bb5 (127.0.98.2:37893), bb5db78b40004e96a9050c5705c55569 (127.0.98.1:44181)
I20260929 14:46:01.551214 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.547099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33257 (local address 127.0.98.1:44181)
0929 14:46:01.547217 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.547221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.547240 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:46:01.547333 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.547335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.547370 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.547439 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:01.547649 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.548358 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.549942 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.550557 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.550660 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.550723 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.551013 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.551089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 14:46:01.551214 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.546810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33849 (local address 127.0.98.2:37893)
0929 14:46:01.546934 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.546939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.546966 (+ 27us) server_negotiation.cc:407] Connection header received
0929 14:46:01.547018 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.547021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.547070 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.547148 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:01.547432 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.548226 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.549942 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.550554 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.550660 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.550723 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.551013 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.551089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:46:01.551231 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.546562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37893 (local address 127.0.98.3:33849)
0929 14:46:01.546690 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.546707 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.546794 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.547179 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.547181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.547201 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:01.547322 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.547328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.548341 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.548344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.549815 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.549825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.550699 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.550703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.550864 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.550903 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.550983 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.551140 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":66}
I20260929 14:46:01.551239 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.546666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.98.3:33257)
0929 14:46:01.547163 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.547175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:01.547258 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:01.547459 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:01.547462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:01.547475 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:01.547556 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.547560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.548489 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.548492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.549815 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:01.549825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.550699 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.550703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:01.550888 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.550905 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:46:01.550983 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 14:46:01.551072 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":94,"thread_start_us":121,"threads_started":1}
I20260929 14:46:01.551988 17724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba7d196a07247408dbddd39b21f9f5f" candidate_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804aecd67c74413da60f507c8c6f7bb5" is_pre_election: true
I20260929 14:46:01.552105 17590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba7d196a07247408dbddd39b21f9f5f" candidate_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5db78b40004e96a9050c5705c55569" is_pre_election: true
I20260929 14:46:01.552354 17590 raft_consensus.cc:2454] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120fc5237adf4a95b74d5fd8d7cce60d in term 0.
I20260929 14:46:01.552356 17724 raft_consensus.cc:2454] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 120fc5237adf4a95b74d5fd8d7cce60d in term 0.
I20260929 14:46:01.552887 17791 leader_election.cc:302] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 120fc5237adf4a95b74d5fd8d7cce60d, 804aecd67c74413da60f507c8c6f7bb5, bb5db78b40004e96a9050c5705c55569; no voters:
I20260929 14:46:01.553213 17929 raft_consensus.cc:2781] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:01.553287 17929 raft_consensus.cc:482] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:01.553318 17929 raft_consensus.cc:3037] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:01.554301 17929 raft_consensus.cc:504] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.554644 17929 leader_election.cc:288] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [CANDIDATE]: Term 1 election: Requested vote from peers 804aecd67c74413da60f507c8c6f7bb5 (127.0.98.2:37893), bb5db78b40004e96a9050c5705c55569 (127.0.98.1:44181)
I20260929 14:46:01.554981 17724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba7d196a07247408dbddd39b21f9f5f" candidate_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804aecd67c74413da60f507c8c6f7bb5"
I20260929 14:46:01.555011 17590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba7d196a07247408dbddd39b21f9f5f" candidate_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5db78b40004e96a9050c5705c55569"
I20260929 14:46:01.555090 17724 raft_consensus.cc:3037] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:01.555159 17590 raft_consensus.cc:3037] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:01.556113 17724 raft_consensus.cc:2454] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120fc5237adf4a95b74d5fd8d7cce60d in term 1.
I20260929 14:46:01.556252 17590 raft_consensus.cc:2454] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 120fc5237adf4a95b74d5fd8d7cce60d in term 1.
I20260929 14:46:01.556492 17794 leader_election.cc:302] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 120fc5237adf4a95b74d5fd8d7cce60d, 804aecd67c74413da60f507c8c6f7bb5; no voters:
I20260929 14:46:01.556728 17929 raft_consensus.cc:2781] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:01.556985 17929 raft_consensus.cc:686] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 1 LEADER]: Becoming Leader. State: Replica: 120fc5237adf4a95b74d5fd8d7cce60d, State: Running, Role: LEADER
I20260929 14:46:01.557348 17929 consensus_queue.cc:237] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } }
I20260929 14:46:01.560516 17440 catalog_manager.cc:5495] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d reported cstate change: term changed from 0 to 1, leader changed from <none> to 120fc5237adf4a95b74d5fd8d7cce60d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "120fc5237adf4a95b74d5fd8d7cce60d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 } health_report { overall_health: UNKNOWN } } }
I20260929 14:46:01.579715 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.575974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.0.98.3:45109)
0929 14:46:01.576225 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.576229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.576242 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:01.576316 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.576319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.576374 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.576462 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:01.576771 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.577342 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.578211 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.578422 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.578519 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.578838 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:01.579286 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:01.579396 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.579476 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.579543 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260929 14:46:01.586005 17724 raft_consensus.cc:1261] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 1 FOLLOWER]: Refusing update from remote peer 120fc5237adf4a95b74d5fd8d7cce60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:01.586005 17590 raft_consensus.cc:1261] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569 [term 1 FOLLOWER]: Refusing update from remote peer 120fc5237adf4a95b74d5fd8d7cce60d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:01.586696 17931 consensus_queue.cc:1035] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "804aecd67c74413da60f507c8c6f7bb5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:01.586889 17929 consensus_queue.cc:1035] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5db78b40004e96a9050c5705c55569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:01.595239 17935 mvcc.cc:205] Tried to move back new op lower bound from 7334679189848756224 to 7334679189741744128. Current Snapshot: MvccSnapshot[applied={T|T < 7334679189848756224}]
I20260929 14:46:01.595785 17936 mvcc.cc:205] Tried to move back new op lower bound from 7334679189848756224 to 7334679189741744128. Current Snapshot: MvccSnapshot[applied={T|T < 7334679189848756224}]
I20260929 14:46:01.597283 17934 mvcc.cc:205] Tried to move back new op lower bound from 7334679189848756224 to 7334679189741744128. Current Snapshot: MvccSnapshot[applied={T|T < 7334679189848756224}]
I20260929 14:46:01.605607 17444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:01.605823 17444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:01.608623 17444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b30928bcd046699b1bfabb369db307: Sending DeleteTablet for 3 replicas of tablet 9ba7d196a07247408dbddd39b21f9f5f
I20260929 14:46:01.609616 17837 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57918
I20260929 14:46:01.609634 17570 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58232
I20260929 14:46:01.610203 17944 tablet_replica.cc:330] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: stopping tablet replica
I20260929 14:46:01.610183 17704 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba7d196a07247408dbddd39b21f9f5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53064
I20260929 14:46:01.610529 17945 tablet_replica.cc:330] T 9ba7d196a07247408dbddd39b21f9f5f P bb5db78b40004e96a9050c5705c55569: stopping tablet replica
I20260929 14:46:01.610816 17946 tablet_replica.cc:330] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: stopping tablet replica
I20260929 14:46:01.611089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17507
I20260929 14:46:01.611208 17946 raft_consensus.cc:2229] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:01.611526 17946 raft_consensus.cc:2258] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:01.612691 17944 raft_consensus.cc:2229] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:46:01.613202 17944 raft_consensus.cc:2258] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:01.613186 17946 ts_tablet_manager.cc:1905] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:01.614821 17944 ts_tablet_manager.cc:1905] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:01.616364 17946 ts_tablet_manager.cc:1918] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:01.616479 17946 log.cc:1196] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/9ba7d196a07247408dbddd39b21f9f5f
I20260929 14:46:01.616806 17946 ts_tablet_manager.cc:1939] T 9ba7d196a07247408dbddd39b21f9f5f P 804aecd67c74413da60f507c8c6f7bb5: Deleting consensus metadata
I20260929 14:46:01.617872 17944 ts_tablet_manager.cc:1918] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:01.618060 17944 log.cc:1196] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/9ba7d196a07247408dbddd39b21f9f5f
I20260929 14:46:01.618378 17432 catalog_manager.cc:4841] TS 804aecd67c74413da60f507c8c6f7bb5 (127.0.98.2:37893): tablet 9ba7d196a07247408dbddd39b21f9f5f (table test-table [id=ad50ab61810841c0ac26ec81bf7b931d]) successfully deleted
I20260929 14:46:01.618453 17944 ts_tablet_manager.cc:1939] T 9ba7d196a07247408dbddd39b21f9f5f P 120fc5237adf4a95b74d5fd8d7cce60d: Deleting consensus metadata
I20260929 14:46:01.620117 17431 catalog_manager.cc:4841] TS 120fc5237adf4a95b74d5fd8d7cce60d (127.0.98.3:45109): tablet 9ba7d196a07247408dbddd39b21f9f5f (table test-table [id=ad50ab61810841c0ac26ec81bf7b931d]) successfully deleted
W20260929 14:46:01.620801 17429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44181 (error 108)
I20260929 14:46:01.621690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17641
I20260929 14:46:01.623400 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.623069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44181 (local address 127.0.0.1:58236)
0929 14:46:01.623238 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:01.623319 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 14:46:01.623588 17429 catalog_manager.cc:4584] TS bb5db78b40004e96a9050c5705c55569 (127.0.98.1:44181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ba7d196a07247408dbddd39b21f9f5f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44181: connect: Connection refused (error 111)
I20260929 14:46:01.627496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17775
I20260929 14:46:01.633031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17413
2026-09-29T14:46:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2272 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T14:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:01Z Disabled control of system clock
I20260929 14:46:01.661597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38543
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:01.774628 17951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:01.774942 17951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:01.774981 17951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:01.779227 17951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:01.779304 17951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:01.779318 17951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:01.779333 17951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:01.779347 17951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:01.784375 17951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:01.785629 17951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:01.787120 17951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:01.795972 17959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:01.795972 17956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:01.795943 17957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:01.796267 17951 server_base.cc:1034] running on GCE node
I20260929 14:46:01.797200 17951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:01.798434 17951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:01.799636 17951 hybrid_clock.cc:648] HybridClock initialized: now 1790693161799600 us; error 55 us; skew 500 ppm
I20260929 14:46:01.801802 17951 webserver.cc:460] Webserver started at http://127.0.98.62:40327/ using document root <none> and password file <none>
I20260929 14:46:01.802430 17951 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:01.802500 17951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:01.802743 17951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:01.804991 17951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "86a9b5a495174464a0f44e0a155e4f80"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "e845f154fbfbc1945f0377138fe13440"
server_key_iv: "97d9a944358993bb8178a6b3bd98e0a2"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.805586 17951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "86a9b5a495174464a0f44e0a155e4f80"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "e845f154fbfbc1945f0377138fe13440"
server_key_iv: "97d9a944358993bb8178a6b3bd98e0a2"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.809710 17951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:01.812441 17965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:01.813917 17951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:01.814129 17951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "86a9b5a495174464a0f44e0a155e4f80"
format_stamp: "Formatted at 2026-09-29 14:46:01 on dist-test-slave-g8hp"
server_key: "e845f154fbfbc1945f0377138fe13440"
server_key_iv: "97d9a944358993bb8178a6b3bd98e0a2"
server_key_version: "encryptionkey@0"
I20260929 14:46:01.814249 17951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:01.832564 17951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:01.833415 17951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:01.833628 17951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:01.841254 17951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38543
I20260929 14:46:01.841275 18017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38543 every 8 connection(s)
I20260929 14:46:01.842314 17951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:01.845738 18018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:01.848065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 17951
I20260929 14:46:01.848193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:01.848448 392 external_mini_cluster.cc:1428] Setting key c26fdb7ed1d1ebbe75295d39a5cb1e6a
I20260929 14:46:01.852041 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: Bootstrap starting.
I20260929 14:46:01.854864 18018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:01.855789 18018 log.cc:824] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:01.858495 18018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: No bootstrap required, opened a new log
I20260929 14:46:01.859316 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:01.850145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.98.62:38543)
0929 14:46:01.850865 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:01.850885 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:01.850932 (+ 47us) server_negotiation.cc:407] Connection header received
0929 14:46:01.851623 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:01.851642 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:01.852096 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:01.852451 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:01.852849 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.853966 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.855350 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:01.856063 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:01.856206 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:01.857222 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:01.857228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:01.857243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:01.857667 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:01.857731 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:01.857753 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:01.857851 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:01.858017 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 14:46:01.858331 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260929 14:46:01.862695 18018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } }
I20260929 14:46:01.862979 18018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:01.863052 18018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a9b5a495174464a0f44e0a155e4f80, State: Initialized, Role: FOLLOWER
I20260929 14:46:01.863497 18018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } }
I20260929 14:46:01.863636 18018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:01.863701 18018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:01.863801 18018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:01.864814 18018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } }
I20260929 14:46:01.865244 18018 leader_election.cc:302] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a9b5a495174464a0f44e0a155e4f80; no voters:
I20260929 14:46:01.865605 18018 leader_election.cc:288] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:01.865754 18023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:01.865962 18023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [term 1 LEADER]: Becoming Leader. State: Replica: 86a9b5a495174464a0f44e0a155e4f80, State: Running, Role: LEADER
I20260929 14:46:01.866290 18023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } }
I20260929 14:46:01.866892 18018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:01.867971 18025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a9b5a495174464a0f44e0a155e4f80. Latest consensus state: current_term: 1 leader_uuid: "86a9b5a495174464a0f44e0a155e4f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } } }
I20260929 14:46:01.868135 18025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:01.868046 18024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a9b5a495174464a0f44e0a155e4f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a9b5a495174464a0f44e0a155e4f80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38543 } } }
I20260929 14:46:01.868373 18024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:01.868665 18032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:01.872020 18032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:01.878080 18032 catalog_manager.cc:1347] Generated new cluster ID: 636e2a0531144519a8b1ec544e20cc38
I20260929 14:46:01.878177 18032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:01.904803 18032 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:01.905867 18032 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:01.919515 18032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: Generated new TSK 0
I20260929 14:46:01.920455 18032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:01.934461 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:02.053222 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:02.053527 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:02.053618 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:02.057900 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:02.057981 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:02.058058 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:02.063207 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:02.064405 18042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:02.066083 18042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:02.076702 18050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:02.076702 18048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:02.076776 18047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:02.076756 18042 server_base.cc:1034] running on GCE node
I20260929 14:46:02.077463 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:02.078223 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:02.079402 18042 hybrid_clock.cc:648] HybridClock initialized: now 1790693162079357 us; error 58 us; skew 500 ppm
I20260929 14:46:02.081521 18042 webserver.cc:460] Webserver started at http://127.0.98.1:46079/ using document root <none> and password file <none>
I20260929 14:46:02.082147 18042 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:02.082222 18042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:02.082425 18042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:02.084820 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5e33f83e611450591adcccff723d29d"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "258b92c2eb4b84a0091a80e82a3479c3"
server_key_iv: "96e9fe7e92e1c3aa1491cb098ea9d31b"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.085489 18042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5e33f83e611450591adcccff723d29d"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "258b92c2eb4b84a0091a80e82a3479c3"
server_key_iv: "96e9fe7e92e1c3aa1491cb098ea9d31b"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.089649 18042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:02.092592 18056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.094048 18042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:46:02.094179 18042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "d5e33f83e611450591adcccff723d29d"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "258b92c2eb4b84a0091a80e82a3479c3"
server_key_iv: "96e9fe7e92e1c3aa1491cb098ea9d31b"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.094308 18042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:02.107007 18042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:02.107767 18042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:02.107969 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:02.108681 18042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:02.109846 18042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:02.109898 18042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.109941 18042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:02.109963 18042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.118993 18042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41311
I20260929 14:46:02.119052 18169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41311 every 8 connection(s)
I20260929 14:46:02.120026 18042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:02.121640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18042
I20260929 14:46:02.121767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:02.122064 392 external_mini_cluster.cc:1428] Setting key 0fa1b8e8c161ae8a2330aac2001e53e9
I20260929 14:46:02.124338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:02.132197 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.122271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57629 (local address 127.0.98.62:38543)
0929 14:46:02.122440 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.122445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.123822 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 14:46:02.125417 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.125421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.125484 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.125582 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:02.127007 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.128150 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.129805 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.130166 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.130296 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.131532 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:02.131536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:02.131540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:02.131560 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:02.131586 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:02.131598 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:02.131702 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.131926 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.132020 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 14:46:02.133472 18172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.122771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38543 (local address 127.0.98.1:57629)
0929 14:46:02.123651 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.123711 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.125141 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.125894 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.125905 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.126037 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:02.126795 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.126815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.128340 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.128344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.129647 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.129659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.130364 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.130367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.130553 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.131197 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:02.131198 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:02.131246 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:02.131331 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:02.131725 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:02.131735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:02.131741 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.131883 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.132251 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":135,"threads_started":1}
I20260929 14:46:02.135064 18170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38543
I20260929 14:46:02.135380 18170 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:02.136080 18170 heartbeater.cc:511] Master 127.0.98.62:38543 requested a full tablet report, sending...
I20260929 14:46:02.137929 17982 ts_manager.cc:194] Registered new tserver with Master: d5e33f83e611450591adcccff723d29d (127.0.98.1:41311)
I20260929 14:46:02.139225 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57629
W20260929 14:46:02.242138 18173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:02.242435 18173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:02.242491 18173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:02.246762 18173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:02.246843 18173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:02.246932 18173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:02.252022 18173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:02.253208 18173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:02.254840 18173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:46:02.265842 18173 server_base.cc:1034] running on GCE node
W20260929 14:46:02.265784 18178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:02.265797 18179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:02.265831 18181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:02.266732 18173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:02.267454 18173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:02.268669 18173 hybrid_clock.cc:648] HybridClock initialized: now 1790693162268632 us; error 50 us; skew 500 ppm
I20260929 14:46:02.270973 18173 webserver.cc:460] Webserver started at http://127.0.98.2:39903/ using document root <none> and password file <none>
I20260929 14:46:02.271584 18173 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:02.271658 18173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:02.271874 18173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:02.274061 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "baca2d9f027b484cb95bf33e894b85fd"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "76ba018af31284e56ed1c8809f297e63"
server_key_iv: "3f231b89d9a19d4b8626b4fb39180072"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.274634 18173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "baca2d9f027b484cb95bf33e894b85fd"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "76ba018af31284e56ed1c8809f297e63"
server_key_iv: "3f231b89d9a19d4b8626b4fb39180072"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.278625 18173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:02.281478 18187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.283174 18173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:02.283313 18173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "baca2d9f027b484cb95bf33e894b85fd"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "76ba018af31284e56ed1c8809f297e63"
server_key_iv: "3f231b89d9a19d4b8626b4fb39180072"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.283442 18173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:02.300525 18173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:02.301311 18173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:02.301501 18173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:02.302117 18173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:02.303103 18173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:02.303150 18173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.303207 18173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:02.303231 18173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.312479 18173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33837
I20260929 14:46:02.312569 18300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33837 every 8 connection(s)
I20260929 14:46:02.313508 18173 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:46:02.322552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18173
I20260929 14:46:02.322669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:46:02.322938 392 external_mini_cluster.cc:1428] Setting key 5c902ba0d938aecf44fbe2aab5035449
I20260929 14:46:02.324011 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.315532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55777 (local address 127.0.98.62:38543)
0929 14:46:02.315684 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.315688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.316915 (+ 1227us) server_negotiation.cc:407] Connection header received
0929 14:46:02.318500 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.318503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.318558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.318682 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:02.319970 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.320535 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.321848 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.322107 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.322207 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.323471 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:02.323475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:02.323477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:02.323495 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:02.323518 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:02.323529 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:02.323600 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.323783 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.323853 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 14:46:02.324991 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.315989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38543 (local address 127.0.98.2:55777)
0929 14:46:02.316712 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.316756 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.318229 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.319064 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.319074 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.319182 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:02.319782 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.319797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.320670 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.320673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.321710 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.321720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.322299 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.322303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.322475 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.323119 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:02.323121 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:02.323167 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:02.323240 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:02.323620 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:02.323628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:02.323637 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.323754 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.324075 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260929 14:46:02.325019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:02.326282 18301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38543
I20260929 14:46:02.326560 18301 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:02.327132 18301 heartbeater.cc:511] Master 127.0.98.62:38543 requested a full tablet report, sending...
I20260929 14:46:02.328321 17982 ts_manager.cc:194] Registered new tserver with Master: baca2d9f027b484cb95bf33e894b85fd (127.0.98.2:33837)
I20260929 14:46:02.329182 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55777
W20260929 14:46:02.439405 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:02.439697 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:02.439736 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:02.443917 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:02.444000 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:02.444057 18304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:02.449410 18304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:02.450500 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:02.451915 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:02.463490 18310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:02.463477 18312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:02.463677 18304 server_base.cc:1034] running on GCE node
W20260929 14:46:02.463478 18309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:02.464347 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:02.465126 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:02.466315 18304 hybrid_clock.cc:648] HybridClock initialized: now 1790693162466281 us; error 56 us; skew 500 ppm
I20260929 14:46:02.468489 18304 webserver.cc:460] Webserver started at http://127.0.98.3:44087/ using document root <none> and password file <none>
I20260929 14:46:02.469144 18304 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:02.469221 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:02.469430 18304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:02.471643 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "56731430d39f49eabbdfa1848a18cace"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "4afbba2bda91de0d7e8621544f815146"
server_key_iv: "5f9a28f36fd5598287491323267211ca"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.472241 18304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "56731430d39f49eabbdfa1848a18cace"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "4afbba2bda91de0d7e8621544f815146"
server_key_iv: "5f9a28f36fd5598287491323267211ca"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.476372 18304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 14:46:02.479298 18318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.480813 18304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:02.481094 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "56731430d39f49eabbdfa1848a18cace"
format_stamp: "Formatted at 2026-09-29 14:46:02 on dist-test-slave-g8hp"
server_key: "4afbba2bda91de0d7e8621544f815146"
server_key_iv: "5f9a28f36fd5598287491323267211ca"
server_key_version: "encryptionkey@0"
I20260929 14:46:02.481228 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:02.510787 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:02.511562 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:02.511758 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:02.512390 18304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:02.513379 18304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:02.513428 18304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.513480 18304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:02.513554 18304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:02.522816 18304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44047
I20260929 14:46:02.522912 18431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44047 every 8 connection(s)
I20260929 14:46:02.524102 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:02.532212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18304
I20260929 14:46:02.532330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:02.532605 392 external_mini_cluster.cc:1428] Setting key 60d19001f0bbf42754ac0b7e65ab7b6c
I20260929 14:46:02.534391 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.526397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44043 (local address 127.0.98.62:38543)
0929 14:46:02.526558 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.526562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.527709 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 14:46:02.529206 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.529210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.529272 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.529350 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:02.530454 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.530982 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.532196 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.532446 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.532544 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.533830 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:02.533834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:02.533837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:02.533852 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:02.533873 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:02.533882 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:02.533938 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.534174 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.534245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 14:46:02.535297 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.526694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38543 (local address 127.0.98.3:44043)
0929 14:46:02.527484 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.527574 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.528961 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.529583 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.529591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.529687 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:02.530289 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.530303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.531126 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.531129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.532056 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.532066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.532614 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.532618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.532806 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.533556 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:02.533558 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:02.533604 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:02.533677 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:02.533969 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:02.533979 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:02.533985 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.534108 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.534367 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":134,"threads_started":1}
I20260929 14:46:02.536656 18432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38543
I20260929 14:46:02.536948 18432 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:02.537571 18432 heartbeater.cc:511] Master 127.0.98.62:38543 requested a full tablet report, sending...
I20260929 14:46:02.538666 17982 ts_manager.cc:194] Registered new tserver with Master: 56731430d39f49eabbdfa1848a18cace (127.0.98.3:44047)
I20260929 14:46:02.539273 17982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44043
I20260929 14:46:02.547152 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:02.552990 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.549198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.0.98.62:38543)
0929 14:46:02.549338 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.549343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.549471 (+ 128us) server_negotiation.cc:407] Connection header received
0929 14:46:02.549613 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.549616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.549660 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.549726 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:02.550014 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.550554 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.551375 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.551587 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.551682 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.552491 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:02.552495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:02.552498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:02.552511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:02.552530 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:02.552546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:02.552606 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.552724 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.552783 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:46:02.556172 17982 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:02.559219 17982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:02.574582 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.568339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41311 (local address 127.0.0.1:60482)
0929 14:46:02.568940 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.568999 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.569136 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.569986 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.569993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.570016 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:02.570163 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.570171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.571516 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.571520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.572808 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.572818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.573969 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.573973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.574110 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574160 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.574274 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.574419 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":89,"threads_started":1}
I20260929 14:46:02.574641 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.569099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33837 (local address 127.0.0.1:50920)
0929 14:46:02.569352 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.569374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.569476 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.570315 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.570317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.570336 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:02.570460 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.570466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.572046 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.572050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.573175 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.573184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.574196 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.574200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.574294 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574354 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.574412 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.574497 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":13,"thread_start_us":89,"threads_started":1}
I20260929 14:46:02.574870 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.569532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.0.98.2:33837)
0929 14:46:02.569848 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.569868 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.569892 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:46:02.569966 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.569971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.570166 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.570636 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:02.570789 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.571898 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.573309 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.574063 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.574372 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574559 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.574613 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.574715 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":99,"threads_started":1}
I20260929 14:46:02.575232 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.568510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.98.1:41311)
0929 14:46:02.569427 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.569439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.569479 (+ 40us) server_negotiation.cc:407] Connection header received
0929 14:46:02.569559 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.569565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.569799 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.569961 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:02.570308 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.571406 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.572999 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.573828 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.573947 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574171 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.574946 (+ 775us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.575062 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":134,"threads_started":1}
I20260929 14:46:02.575255 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.568510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.0.1:33670)
0929 14:46:02.569071 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.569082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.569142 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.570895 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.570898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.570916 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:02.571012 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.571018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.572362 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.572365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.573224 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.573233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.574286 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.574290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.574972 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574997 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.575067 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.575135 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":21,"thread_start_us":109,"threads_started":1}
I20260929 14:46:02.575671 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.569557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.0.98.3:44047)
0929 14:46:02.570290 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.570306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.570328 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:46:02.570406 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.570410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.570602 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.570746 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:02.571157 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.572239 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.573367 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.574180 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.574294 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.574481 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.575404 (+ 923us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.575517 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":147,"threads_started":1}
I20260929 14:46:02.578042 18235 tablet_service.cc:1467] Processing CreateTablet for tablet d1a0aa9681324cf099db66c341707394 (DEFAULT_TABLE table=test-table [id=40b4b407314643339eca4013e19e5dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:02.578038 18104 tablet_service.cc:1467] Processing CreateTablet for tablet d1a0aa9681324cf099db66c341707394 (DEFAULT_TABLE table=test-table [id=40b4b407314643339eca4013e19e5dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:02.579077 18104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a0aa9681324cf099db66c341707394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:02.579085 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a0aa9681324cf099db66c341707394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:02.583365 18366 tablet_service.cc:1467] Processing CreateTablet for tablet d1a0aa9681324cf099db66c341707394 (DEFAULT_TABLE table=test-table [id=40b4b407314643339eca4013e19e5dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:02.584635 18366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a0aa9681324cf099db66c341707394. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:02.585923 18449 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Bootstrap starting.
I20260929 14:46:02.587334 18450 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Bootstrap starting.
I20260929 14:46:02.588547 18449 tablet_bootstrap.cc:654] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:02.589596 18449 log.cc:824] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:02.589763 18450 tablet_bootstrap.cc:654] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:02.590656 18450 log.cc:824] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:02.591635 18449 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: No bootstrap required, opened a new log
I20260929 14:46:02.591861 18449 ts_tablet_manager.cc:1397] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:46:02.592911 18453 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Bootstrap starting.
I20260929 14:46:02.593622 18450 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: No bootstrap required, opened a new log
I20260929 14:46:02.593796 18450 ts_tablet_manager.cc:1397] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:46:02.596707 18453 tablet_bootstrap.cc:654] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:02.596467 18449 raft_consensus.cc:348] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.596894 18449 raft_consensus.cc:374] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:02.596951 18449 raft_consensus.cc:729] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e33f83e611450591adcccff723d29d, State: Initialized, Role: FOLLOWER
I20260929 14:46:02.596918 18450 raft_consensus.cc:348] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.597210 18450 raft_consensus.cc:374] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:02.597266 18450 raft_consensus.cc:729] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baca2d9f027b484cb95bf33e894b85fd, State: Initialized, Role: FOLLOWER
I20260929 14:46:02.597448 18449 consensus_queue.cc:260] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.597672 18450 consensus_queue.cc:260] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.597855 18453 log.cc:824] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:02.598433 18170 heartbeater.cc:503] Master 127.0.98.62:38543 was elected leader, sending a full tablet report...
I20260929 14:46:02.598599 18449 ts_tablet_manager.cc:1428] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:46:02.598706 18450 ts_tablet_manager.cc:1428] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:46:02.599116 18301 heartbeater.cc:503] Master 127.0.98.62:38543 was elected leader, sending a full tablet report...
I20260929 14:46:02.600962 18453 tablet_bootstrap.cc:492] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: No bootstrap required, opened a new log
I20260929 14:46:02.601182 18453 ts_tablet_manager.cc:1397] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 14:46:02.605034 18453 raft_consensus.cc:348] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.605310 18453 raft_consensus.cc:374] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:02.605369 18453 raft_consensus.cc:729] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56731430d39f49eabbdfa1848a18cace, State: Initialized, Role: FOLLOWER
I20260929 14:46:02.605826 18453 consensus_queue.cc:260] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.606554 18432 heartbeater.cc:503] Master 127.0.98.62:38543 was elected leader, sending a full tablet report...
I20260929 14:46:02.606799 18453 ts_tablet_manager.cc:1428] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260929 14:46:02.621315 18171 tablet.cc:2369] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:02.703488 18454 raft_consensus.cc:482] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:02.703708 18454 raft_consensus.cc:504] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.704972 18454 leader_election.cc:288] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56731430d39f49eabbdfa1848a18cace (127.0.98.3:44047), baca2d9f027b484cb95bf33e894b85fd (127.0.98.2:33837)
I20260929 14:46:02.708984 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.705392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49401 (local address 127.0.98.3:44047)
0929 14:46:02.705589 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.705594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.705727 (+ 133us) server_negotiation.cc:407] Connection header received
0929 14:46:02.705890 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.705893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.705934 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.706003 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:02.706367 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.706860 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.707963 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.708405 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.708507 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.708569 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.708760 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.708830 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260929 14:46:02.709012 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.705261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44047 (local address 127.0.98.1:49401)
0929 14:46:02.705632 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.705647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.705775 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.706115 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.706118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.706137 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:02.706249 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.706253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.706981 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.706984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.707832 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.707841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.708542 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.708546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.708643 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.708662 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.708726 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.708802 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":27,"thread_start_us":90,"threads_started":1}
I20260929 14:46:02.709118 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.705368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53795 (local address 127.0.98.2:33837)
0929 14:46:02.705501 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.705506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.706072 (+ 566us) server_negotiation.cc:407] Connection header received
0929 14:46:02.706199 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.706202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.706249 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.706315 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:02.706608 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.707087 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.708134 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.708538 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.708630 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.708689 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.708915 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.708987 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:46:02.709340 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.705261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33837 (local address 127.0.98.1:53795)
0929 14:46:02.705933 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.705947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:02.706081 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:02.706345 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:02.706347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:02.706363 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:02.706492 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.706498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.707200 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.707203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.707998 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:02.708007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.708690 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.708693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:02.708784 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.708801 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:46:02.708907 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:46:02.708990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260929 14:46:02.709667 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a0aa9681324cf099db66c341707394" candidate_uuid: "d5e33f83e611450591adcccff723d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56731430d39f49eabbdfa1848a18cace" is_pre_election: true
I20260929 14:46:02.710019 18386 raft_consensus.cc:2454] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5e33f83e611450591adcccff723d29d in term 0.
I20260929 14:46:02.710009 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a0aa9681324cf099db66c341707394" candidate_uuid: "d5e33f83e611450591adcccff723d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baca2d9f027b484cb95bf33e894b85fd" is_pre_election: true
I20260929 14:46:02.710266 18255 raft_consensus.cc:2454] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5e33f83e611450591adcccff723d29d in term 0.
I20260929 14:46:02.710549 18060 leader_election.cc:302] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56731430d39f49eabbdfa1848a18cace, d5e33f83e611450591adcccff723d29d; no voters:
I20260929 14:46:02.710851 18454 raft_consensus.cc:2781] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:02.710948 18454 raft_consensus.cc:482] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:02.710979 18454 raft_consensus.cc:3037] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:02.712266 18454 raft_consensus.cc:504] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.712657 18454 leader_election.cc:288] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [CANDIDATE]: Term 1 election: Requested vote from peers 56731430d39f49eabbdfa1848a18cace (127.0.98.3:44047), baca2d9f027b484cb95bf33e894b85fd (127.0.98.2:33837)
I20260929 14:46:02.713124 18255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a0aa9681324cf099db66c341707394" candidate_uuid: "d5e33f83e611450591adcccff723d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baca2d9f027b484cb95bf33e894b85fd"
I20260929 14:46:02.713238 18386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a0aa9681324cf099db66c341707394" candidate_uuid: "d5e33f83e611450591adcccff723d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56731430d39f49eabbdfa1848a18cace"
I20260929 14:46:02.713272 18255 raft_consensus.cc:3037] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:02.713372 18386 raft_consensus.cc:3037] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:02.714627 18386 raft_consensus.cc:2454] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5e33f83e611450591adcccff723d29d in term 1.
I20260929 14:46:02.714627 18255 raft_consensus.cc:2454] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5e33f83e611450591adcccff723d29d in term 1.
I20260929 14:46:02.715040 18060 leader_election.cc:302] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56731430d39f49eabbdfa1848a18cace, d5e33f83e611450591adcccff723d29d; no voters:
I20260929 14:46:02.715234 18454 raft_consensus.cc:2781] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:02.715461 18454 raft_consensus.cc:686] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [term 1 LEADER]: Becoming Leader. State: Replica: d5e33f83e611450591adcccff723d29d, State: Running, Role: LEADER
I20260929 14:46:02.715847 18454 consensus_queue.cc:237] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } }
I20260929 14:46:02.719864 17978 catalog_manager.cc:5495] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d reported cstate change: term changed from 0 to 1, leader changed from <none> to d5e33f83e611450591adcccff723d29d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5e33f83e611450591adcccff723d29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e33f83e611450591adcccff723d29d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 } health_report { overall_health: UNKNOWN } } }
I20260929 14:46:02.734341 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.730634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.98.1:41311)
0929 14:46:02.730778 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:02.730782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:02.730796 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:46:02.730880 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:02.730882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:02.730940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:02.731034 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:02.731364 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.731926 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.732800 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:02.733084 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:02.733177 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:02.733492 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:02.733945 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:02.734058 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:02.734136 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 14:46:02.734197 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:46:02.741277 18255 raft_consensus.cc:1261] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 1 FOLLOWER]: Refusing update from remote peer d5e33f83e611450591adcccff723d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:02.741276 18386 raft_consensus.cc:1261] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 1 FOLLOWER]: Refusing update from remote peer d5e33f83e611450591adcccff723d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:02.742277 18460 consensus_queue.cc:1035] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56731430d39f49eabbdfa1848a18cace" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:02.742508 18454 consensus_queue.cc:1035] T d1a0aa9681324cf099db66c341707394 P d5e33f83e611450591adcccff723d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "baca2d9f027b484cb95bf33e894b85fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:02.754927 18465 mvcc.cc:205] Tried to move back new op lower bound from 7334679194578370560 to 7334679194487275520. Current Snapshot: MvccSnapshot[applied={T|T < 7334679194578370560}]
I20260929 14:46:02.754906 18464 mvcc.cc:205] Tried to move back new op lower bound from 7334679194578370560 to 7334679194487275520. Current Snapshot: MvccSnapshot[applied={T|T < 7334679194578370560 or (T in {7334679194578370560})}]
I20260929 14:46:02.762423 17978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:02.762645 17978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:02.765594 17978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86a9b5a495174464a0f44e0a155e4f80: Sending DeleteTablet for 3 replicas of tablet d1a0aa9681324cf099db66c341707394
I20260929 14:46:02.766605 18366 tablet_service.cc:1514] Processing DeleteTablet for tablet d1a0aa9681324cf099db66c341707394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:02 UTC) from {username='slave'} at 127.0.0.1:33670
I20260929 14:46:02.766683 18235 tablet_service.cc:1514] Processing DeleteTablet for tablet d1a0aa9681324cf099db66c341707394 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:02 UTC) from {username='slave'} at 127.0.0.1:50920
I20260929 14:46:02.767179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18042
I20260929 14:46:02.767195 18474 tablet_replica.cc:330] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: stopping tablet replica
I20260929 14:46:02.767573 18474 raft_consensus.cc:2229] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:02.767941 18474 raft_consensus.cc:2258] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:02.768384 18473 tablet_replica.cc:330] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: stopping tablet replica
I20260929 14:46:02.768806 18473 raft_consensus.cc:2229] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:02.769227 18473 raft_consensus.cc:2258] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:02.769739 18474 ts_tablet_manager.cc:1905] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:02.770989 18473 ts_tablet_manager.cc:1905] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:02.772599 18474 ts_tablet_manager.cc:1918] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:02.772706 18474 log.cc:1196] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/d1a0aa9681324cf099db66c341707394
I20260929 14:46:02.773025 18474 ts_tablet_manager.cc:1939] T d1a0aa9681324cf099db66c341707394 P baca2d9f027b484cb95bf33e894b85fd: Deleting consensus metadata
I20260929 14:46:02.773325 18473 ts_tablet_manager.cc:1918] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:02.773411 18473 log.cc:1196] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/d1a0aa9681324cf099db66c341707394
I20260929 14:46:02.773654 18473 ts_tablet_manager.cc:1939] T d1a0aa9681324cf099db66c341707394 P 56731430d39f49eabbdfa1848a18cace: Deleting consensus metadata
I20260929 14:46:02.774494 17968 catalog_manager.cc:4841] TS baca2d9f027b484cb95bf33e894b85fd (127.0.98.2:33837): tablet d1a0aa9681324cf099db66c341707394 (table test-table [id=40b4b407314643339eca4013e19e5dc7]) successfully deleted
I20260929 14:46:02.774730 17969 catalog_manager.cc:4841] TS 56731430d39f49eabbdfa1848a18cace (127.0.98.3:44047): tablet d1a0aa9681324cf099db66c341707394 (table test-table [id=40b4b407314643339eca4013e19e5dc7]) successfully deleted
W20260929 14:46:02.775134 17966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41311 (error 108)
I20260929 14:46:02.775363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18173
I20260929 14:46:02.776496 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:02.776160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41311 (local address 127.0.0.1:60496)
0929 14:46:02.776333 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:02.776410 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 14:46:02.776667 17966 catalog_manager.cc:4584] TS d5e33f83e611450591adcccff723d29d (127.0.98.1:41311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1a0aa9681324cf099db66c341707394: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41311: connect: Connection refused (error 111)
I20260929 14:46:02.780787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18304
I20260929 14:46:02.785722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 17951
2026-09-29T14:46:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:02 dist-test-slave-g8hp krb5kdc[18478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:02 dist-test-slave-g8hp krb5kdc[18478](info): set up 2 sockets
Sep 29 14:46:02 dist-test-slave-g8hp krb5kdc[18478](info): commencing operation
krb5kdc: starting...
W20260929 14:46:03.760648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:03 dist-test-slave-g8hp krb5kdc[18478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:03.890543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34243
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:04.006013 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:04.006320 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:04.006362 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:04.010710 18491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:04.010795 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:04.010815 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:04.010833 18491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:04.010847 18491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:04.016911 18491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:04.018443 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:04.019652 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:04.029359 18497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.029320 18499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.029346 18496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.029671 18491 server_base.cc:1034] running on GCE node
I20260929 14:46:04.030491 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:04.031760 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:04.033016 18491 hybrid_clock.cc:648] HybridClock initialized: now 1790693164032992 us; error 41 us; skew 500 ppm
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:04.036098 18491 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:04.037194 18491 webserver.cc:460] Webserver started at http://127.0.98.62:34625/ using document root <none> and password file <none>
I20260929 14:46:04.037776 18491 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:04.037847 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:04.038056 18491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:04.040200 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "10851c2788f8426e98471813fc1701e8"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "887bc8f0899d747b4d07ffbcd0192dde"
server_key_iv: "f4b3c693464dee60720d02c96a6a7763"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.040764 18491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "10851c2788f8426e98471813fc1701e8"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "887bc8f0899d747b4d07ffbcd0192dde"
server_key_iv: "f4b3c693464dee60720d02c96a6a7763"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.044641 18491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:04.047571 18506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.049225 18491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:04.049361 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "10851c2788f8426e98471813fc1701e8"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "887bc8f0899d747b4d07ffbcd0192dde"
server_key_iv: "f4b3c693464dee60720d02c96a6a7763"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.049490 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:04.066102 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:04.069535 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:04.069732 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:04.076745 18491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34243
I20260929 14:46:04.076812 18558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34243 every 8 connection(s)
I20260929 14:46:04.077806 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:04.080760 18559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:04.086654 18559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: Bootstrap starting.
I20260929 14:46:04.088063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18491
I20260929 14:46:04.088228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:04.088503 392 external_mini_cluster.cc:1428] Setting key a251e2daa3b75e51672dd596fa3307f4
I20260929 14:46:04.089859 18559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:04.090626 18559 log.cc:824] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:04.092983 18559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: No bootstrap required, opened a new log
I20260929 14:46:04.097707 18559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } }
I20260929 14:46:04.097965 18559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:04.098007 18559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10851c2788f8426e98471813fc1701e8, State: Initialized, Role: FOLLOWER
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:04.098557 18559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } }
I20260929 14:46:04.098691 18559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:04.098757 18559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:04.098851 18559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:04.099833 18559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } }
I20260929 14:46:04.100175 18559 leader_election.cc:302] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10851c2788f8426e98471813fc1701e8; no voters:
I20260929 14:46:04.100494 18559 leader_election.cc:288] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:04.100615 18564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:04.100775 18564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [term 1 LEADER]: Becoming Leader. State: Replica: 10851c2788f8426e98471813fc1701e8, State: Running, Role: LEADER
I20260929 14:46:04.101150 18564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } }
I20260929 14:46:04.101907 18559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:04.102780 18565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10851c2788f8426e98471813fc1701e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } } }
I20260929 14:46:04.102957 18565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:04.103261 18566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10851c2788f8426e98471813fc1701e8. Latest consensus state: current_term: 1 leader_uuid: "10851c2788f8426e98471813fc1701e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10851c2788f8426e98471813fc1701e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34243 } } }
I20260929 14:46:04.103320 18573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:04.103353 18566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:04.104341 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.089725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.62:34243)
0929 14:46:04.090173 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.090189 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.090224 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:46:04.091008 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.091034 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.091476 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.091799 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:04.092790 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.093888 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.095113 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.095670 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.095805 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.098756 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:04.098764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:04.098785 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:04.098825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:04.100608 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.101201 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.101208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.101214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.101380 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.101667 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.101671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.101673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.102138 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:04.102346 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.102633 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.103132 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":139,"threads_started":1}
I20260929 14:46:04.108485 18573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:04.115516 18573 catalog_manager.cc:1347] Generated new cluster ID: 31fbf8c5706b4358a448b776f2634b1c
I20260929 14:46:04.115614 18573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:04.139551 18573 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:04.140687 18573 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:04.149833 18573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: Generated new TSK 0
I20260929 14:46:04.150769 18573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:46:04.196991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:04.313020 18585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:04.313324 18585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:04.313370 18585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:04.317814 18585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:04.317901 18585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:04.317965 18585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:04.324229 18585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:04.325481 18585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:04.326548 18585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:04.337934 18590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.337997 18593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.338034 18585 server_base.cc:1034] running on GCE node
W20260929 14:46:04.337934 18591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.338718 18585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:04.339438 18585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:04.340626 18585 hybrid_clock.cc:648] HybridClock initialized: now 1790693164340594 us; error 49 us; skew 500 ppm
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:04.343737 18585 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:46:04.344830 18585 webserver.cc:460] Webserver started at http://127.0.98.1:40577/ using document root <none> and password file <none>
I20260929 14:46:04.345435 18585 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:04.345510 18585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:04.345712 18585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:04.347872 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "15029e3dbf41cb550695ff7a62aec5cb"
server_key_iv: "e9ed6602b287bbaaa94fc324fdddbd22"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.348479 18585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "15029e3dbf41cb550695ff7a62aec5cb"
server_key_iv: "e9ed6602b287bbaaa94fc324fdddbd22"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.352491 18585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:04.355389 18600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.357016 18585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:04.357144 18585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "15029e3dbf41cb550695ff7a62aec5cb"
server_key_iv: "e9ed6602b287bbaaa94fc324fdddbd22"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.357261 18585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:04.368595 18585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:04.371889 18585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:04.372078 18585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:04.372681 18585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:04.373607 18585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:04.373677 18585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.373754 18585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:04.373780 18585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.382982 18585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43109
I20260929 14:46:04.383028 18713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43109 every 8 connection(s)
I20260929 14:46:04.383963 18585 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:04.393843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18585
I20260929 14:46:04.393965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:04.394253 392 external_mini_cluster.cc:1428] Setting key 3f28b417956be17f2cbfd5504884efe1
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:04.399438 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.386189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35151 (local address 127.0.98.62:34243)
0929 14:46:04.386382 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.386386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.387291 (+ 905us) server_negotiation.cc:407] Connection header received
0929 14:46:04.388172 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.388175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.388242 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.388321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:04.389895 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.390477 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.391825 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.392101 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.392197 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.395039 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:04.395043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:04.395045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:04.395076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:04.396837 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.397512 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.397516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.397518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.397613 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.398101 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.398104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.398105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.398330 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:04.398412 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.399165 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.399288 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 14:46:04.400341 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.386561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34243 (local address 127.0.98.1:35151)
0929 14:46:04.387105 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.387153 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.387958 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.388501 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.388508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.389085 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:04.389615 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.389631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.390628 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.390631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.391583 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.391595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.392274 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.392277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.392440 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.393096 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:04.393114 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:04.394816 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:04.397039 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.397046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.397063 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.397405 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.397750 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.397753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.397756 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.397996 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.398440 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:04.398449 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:04.398837 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.399078 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.399450 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
I20260929 14:46:04.401724 18714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34243
I20260929 14:46:04.402148 18714 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:04.402928 18714 heartbeater.cc:511] Master 127.0.98.62:34243 requested a full tablet report, sending...
I20260929 14:46:04.404834 18523 ts_manager.cc:194] Registered new tserver with Master: 1b30d5cbf65746c8a2fba0d7f28ffe7e (127.0.98.1:43109)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:46:04.406194 18523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:46:04.429347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:04.542881 18719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:04.543180 18719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:04.543224 18719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:04.547465 18719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:04.547540 18719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:04.547602 18719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:04.553236 18719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:04.554407 18719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:04.555444 18719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:04.565507 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.565483 18724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.565554 18719 server_base.cc:1034] running on GCE node
W20260929 14:46:04.565491 18727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.566183 18719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:04.566882 18719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:04.568063 18719 hybrid_clock.cc:648] HybridClock initialized: now 1790693164568040 us; error 36 us; skew 500 ppm
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:04.571395 18719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:46:04.572491 18719 webserver.cc:460] Webserver started at http://127.0.98.2:40357/ using document root <none> and password file <none>
I20260929 14:46:04.573130 18719 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:04.573210 18719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:04.573434 18719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:04.575624 18719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff731f0fb869419ebda9b4f11b24f9ab"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "5e936f1f4542a709a8cca83cebad3367"
server_key_iv: "fd6fe20558ecce6124eb832b8000ec6b"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.576231 18719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff731f0fb869419ebda9b4f11b24f9ab"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "5e936f1f4542a709a8cca83cebad3367"
server_key_iv: "fd6fe20558ecce6124eb832b8000ec6b"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.580574 18719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:04.583734 18734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.585793 18719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:04.586155 18719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "ff731f0fb869419ebda9b4f11b24f9ab"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "5e936f1f4542a709a8cca83cebad3367"
server_key_iv: "fd6fe20558ecce6124eb832b8000ec6b"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.586315 18719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:04.616667 18719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:04.620446 18719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:04.620647 18719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:04.621325 18719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:04.622366 18719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:04.622426 18719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.622479 18719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:04.622514 18719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.632781 18719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35909
I20260929 14:46:04.632964 18847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35909 every 8 connection(s)
I20260929 14:46:04.633787 18719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:46:04.636411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18719
I20260929 14:46:04.636536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:46:04.636891 392 external_mini_cluster.cc:1428] Setting key 74b945356f688d2382e68216c187194d
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 14:46:04.650307 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.635872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55691 (local address 127.0.98.62:34243)
0929 14:46:04.636032 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.636037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.637399 (+ 1362us) server_negotiation.cc:407] Connection header received
0929 14:46:04.638836 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.638839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.638897 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.638980 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:04.640484 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.641163 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.642935 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.643161 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.643271 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.646199 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:04.646202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:04.646204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:04.646235 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:04.647667 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.648393 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.648397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.648398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.648489 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.648986 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.648990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.648992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.649217 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:04.649299 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.649998 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.650136 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 14:46:04.651270 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.636443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34243 (local address 127.0.98.2:55691)
0929 14:46:04.637206 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.637261 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.638591 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.639142 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.639150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.639522 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:04.640306 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.640326 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.641309 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.641313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.642740 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.642753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.643374 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.643378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.643559 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.644170 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:04.644187 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:04.645949 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:04.647938 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.647944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.647959 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.648270 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.648600 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.648603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.648605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.648793 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.649316 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:04.649321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:04.649607 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.649894 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.650308 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":163,"threads_started":1}
I20260929 14:46:04.652616 18848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34243
I20260929 14:46:04.652927 18848 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:04.653540 18848 heartbeater.cc:511] Master 127.0.98.62:34243 requested a full tablet report, sending...
I20260929 14:46:04.654603 18523 ts_manager.cc:194] Registered new tserver with Master: ff731f0fb869419ebda9b4f11b24f9ab (127.0.98.2:35909)
I20260929 14:46:04.655184 18523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:46:04.668920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:04.783409 18853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:04.783713 18853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:04.783758 18853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:04.787899 18853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:04.787983 18853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:04.788046 18853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:04.793705 18853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:04.794782 18853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:04.795779 18853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:04.805931 18861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.805943 18859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:04.806169 18858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:04.806440 18853 server_base.cc:1034] running on GCE node
I20260929 14:46:04.806881 18853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:04.807543 18853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:04.808724 18853 hybrid_clock.cc:648] HybridClock initialized: now 1790693164808695 us; error 76 us; skew 500 ppm
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:04.811687 18853 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:46:04.812713 18853 webserver.cc:460] Webserver started at http://127.0.98.3:36411/ using document root <none> and password file <none>
I20260929 14:46:04.813341 18853 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:04.813416 18853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:04.813616 18853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:04.815863 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0b50f5bc155473eb79bb3e8dba1573e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "7b43a1184d89bf0113fa71b36154946e"
server_key_iv: "40cb2b5415a1c825f6450bcacd72e766"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.816495 18853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0b50f5bc155473eb79bb3e8dba1573e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "7b43a1184d89bf0113fa71b36154946e"
server_key_iv: "40cb2b5415a1c825f6450bcacd72e766"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.820477 18853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:46:04.823181 18868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.824662 18853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:04.824785 18853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "b0b50f5bc155473eb79bb3e8dba1573e"
format_stamp: "Formatted at 2026-09-29 14:46:04 on dist-test-slave-g8hp"
server_key: "7b43a1184d89bf0113fa71b36154946e"
server_key_iv: "40cb2b5415a1c825f6450bcacd72e766"
server_key_version: "encryptionkey@0"
I20260929 14:46:04.824983 18853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:04.855208 18853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:04.858680 18853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:04.858882 18853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:04.859512 18853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:04.860364 18853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:04.860412 18853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.860466 18853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:04.860494 18853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:04.869725 18853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34943
I20260929 14:46:04.869757 18981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34943 every 8 connection(s)
I20260929 14:46:04.870700 18853 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:04.876257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 18853
I20260929 14:46:04.876391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:04.876652 392 external_mini_cluster.cc:1428] Setting key 51698b3267a3952b39d05b994b7ebe44
Sep 29 14:46:04 dist-test-slave-g8hp krb5kdc[18478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:04.885205 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.872730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52295 (local address 127.0.98.62:34243)
0929 14:46:04.872898 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.872903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.873847 (+ 944us) server_negotiation.cc:407] Connection header received
0929 14:46:04.874819 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.874822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.874960 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.875106 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:04.876297 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.876927 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.878391 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.878703 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.878806 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.881402 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:04.881406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:04.881408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:04.881433 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:04.883011 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.883551 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.883553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.883555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.883637 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.884000 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.884003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.884004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.884192 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:04.884268 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.884940 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.885065 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:46:04.886145 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.873199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34243 (local address 127.0.98.3:52295)
0929 14:46:04.873679 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.873724 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.874602 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.875279 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.875287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.875644 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:04.876151 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.876165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.877054 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.877064 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.878275 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.878284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.878900 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.878905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.879089 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.879629 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:04.879645 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:04.881227 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:04.883129 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.883134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.883154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.883446 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.883754 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:04.883757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:04.883759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:04.883916 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:04.884306 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:04.884313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:04.884617 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.884798 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.885203 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260929 14:46:04.887737 18982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34243
I20260929 14:46:04.888064 18982 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:04.888622 18982 heartbeater.cc:511] Master 127.0.98.62:34243 requested a full tablet report, sending...
I20260929 14:46:04.889904 18523 ts_manager.cc:194] Registered new tserver with Master: b0b50f5bc155473eb79bb3e8dba1573e (127.0.98.3:34943)
I20260929 14:46:04.890610 18523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52295
I20260929 14:46:04.890875 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:04.901190 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.893148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.0.98.62:34243)
0929 14:46:04.893277 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.893281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.893298 (+ 17us) server_negotiation.cc:407] Connection header received
0929 14:46:04.893405 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.893407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.893448 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.893511 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:04.894301 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.894851 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.895729 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.895962 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.896051 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.897965 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:04.897968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:04.897970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:04.897997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:04.899343 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.899903 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.899906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.899908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.899985 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:04.900333 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:04.900335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:04.900336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:04.900541 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:04.900650 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.900936 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.901024 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:46:04.904428 18523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:04.907416 18523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:04.922187 18994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35909 (local address 127.0.0.1:35566)
0929 14:46:04.916915 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.916970 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.917105 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.917889 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.917894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.917915 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:04.918054 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.918060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919603 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.919607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.920551 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.920570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921562 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.921688 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921741 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.921867 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.922006 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":79,"threads_started":1}
I20260929 14:46:04.922214 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.2:35909)
0929 14:46:04.917392 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.917403 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.917426 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:46:04.917504 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.917510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.917731 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.918207 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:04.918371 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919439 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.920688 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921431 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921532 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921693 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.921940 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.922051 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":154,"threads_started":1}
I20260929 14:46:04.922546 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.0.98.1:43109)
0929 14:46:04.917173 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.917180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.917489 (+ 309us) server_negotiation.cc:407] Connection header received
0929 14:46:04.917556 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.917561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.917723 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.917866 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:04.918215 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919394 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.920819 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921597 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921683 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921792 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.922275 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.922385 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":104,"threads_started":1}
I20260929 14:46:04.922207 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43109 (local address 127.0.0.1:49768)
0929 14:46:04.917309 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.917321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.917405 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.918002 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.918005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.918020 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:04.918094 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.918100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919520 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.919523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.920699 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.920707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921770 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.921922 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921943 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.922017 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.922079 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":38,"thread_start_us":106,"threads_started":1}
I20260929 14:46:04.922776 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.0.98.3:34943)
0929 14:46:04.917427 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:04.917439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:04.917462 (+ 23us) server_negotiation.cc:407] Connection header received
0929 14:46:04.917537 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:04.917541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:04.917733 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:04.917882 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:04.918277 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919018 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.920603 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921487 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921593 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921740 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:04.922425 (+ 685us) server_negotiation.cc:299] Negotiation successful
0929 14:46:04.922528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260929 14:46:04.922212 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:04.916276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34943 (local address 127.0.0.1:45752)
0929 14:46:04.916914 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:04.916970 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:04.917104 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:04.918050 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:04.918053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:04.918071 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:04.918158 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.918165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.919149 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.919153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.920473 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:04.920484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:04.921804 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:04.921808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:04.921902 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:04.921930 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 14:46:04.921995 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:46:04.922066 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":3,"spinlock_wait_cycles":32512,"thread_start_us":98,"threads_started":1}
I20260929 14:46:04.925338 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b (DEFAULT_TABLE table=test-table [id=63af1207c38f45e2bdf54d3614b24dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:04.925597 18648 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b (DEFAULT_TABLE table=test-table [id=63af1207c38f45e2bdf54d3614b24dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:04.925598 18916 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b (DEFAULT_TABLE table=test-table [id=63af1207c38f45e2bdf54d3614b24dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:04.926373 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:04.926640 18916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:04.926640 18648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:04.932904 18999 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Bootstrap starting.
I20260929 14:46:04.935155 19000 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Bootstrap starting.
I20260929 14:46:04.935401 19001 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Bootstrap starting.
I20260929 14:46:04.935559 18999 tablet_bootstrap.cc:654] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:04.936893 18999 log.cc:824] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:04.937255 19000 tablet_bootstrap.cc:654] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:04.937570 19001 tablet_bootstrap.cc:654] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:04.937965 19000 log.cc:824] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:04.938665 19001 log.cc:824] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:04.939141 18999 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: No bootstrap required, opened a new log
I20260929 14:46:04.939347 18999 ts_tablet_manager.cc:1397] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:46:04.939865 19000 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: No bootstrap required, opened a new log
I20260929 14:46:04.940044 19000 ts_tablet_manager.cc:1397] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 14:46:04.940498 19001 tablet_bootstrap.cc:492] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: No bootstrap required, opened a new log
I20260929 14:46:04.940662 19001 ts_tablet_manager.cc:1397] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:46:04.943327 18999 raft_consensus.cc:348] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.943437 19000 raft_consensus.cc:348] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.943645 18999 raft_consensus.cc:374] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:04.943678 19000 raft_consensus.cc:374] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:04.943712 18999 raft_consensus.cc:729] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff731f0fb869419ebda9b4f11b24f9ab, State: Initialized, Role: FOLLOWER
I20260929 14:46:04.943727 19000 raft_consensus.cc:729] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b30d5cbf65746c8a2fba0d7f28ffe7e, State: Initialized, Role: FOLLOWER
I20260929 14:46:04.944281 19000 consensus_queue.cc:260] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.944288 18999 consensus_queue.cc:260] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.944396 19001 raft_consensus.cc:348] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.944723 19001 raft_consensus.cc:374] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:04.944797 19001 raft_consensus.cc:729] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b50f5bc155473eb79bb3e8dba1573e, State: Initialized, Role: FOLLOWER
I20260929 14:46:04.945505 19000 ts_tablet_manager.cc:1428] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:46:04.945518 18999 ts_tablet_manager.cc:1428] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:46:04.945462 19001 consensus_queue.cc:260] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:04.945874 18848 heartbeater.cc:503] Master 127.0.98.62:34243 was elected leader, sending a full tablet report...
I20260929 14:46:04.945873 18714 heartbeater.cc:503] Master 127.0.98.62:34243 was elected leader, sending a full tablet report...
I20260929 14:46:04.946504 19001 ts_tablet_manager.cc:1428] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:46:04.946837 18982 heartbeater.cc:503] Master 127.0.98.62:34243 was elected leader, sending a full tablet report...
I20260929 14:46:05.119195 19007 raft_consensus.cc:482] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:05.119459 19007 raft_consensus.cc:504] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:05.121209 19007 leader_election.cc:288] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff731f0fb869419ebda9b4f11b24f9ab (127.0.98.2:35909), 1b30d5cbf65746c8a2fba0d7f28ffe7e (127.0.98.1:43109)
W20260929 14:46:05.121681 18983 tablet.cc:2369] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:05.125695 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.121415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35909 (local address 127.0.98.3:36193)
0929 14:46:05.121613 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:05.121629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:05.121739 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:05.122135 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:05.122139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:05.122155 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:05.122274 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.122281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.123246 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.123249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.124254 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.124263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.125233 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.125237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.125347 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.125373 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 14:46:05.125462 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 14:46:05.125540 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 14:46:05.125746 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.121479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36193 (local address 127.0.98.2:35909)
0929 14:46:05.121664 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.121671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.121697 (+ 26us) server_negotiation.cc:407] Connection header received
0929 14:46:05.121894 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.121898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.121949 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.122015 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:05.122392 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.123107 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.124412 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.125038 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.125124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.125184 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.125522 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.125613 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 14:46:05.125993 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.121425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43109 (local address 127.0.98.3:48575)
0929 14:46:05.122160 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:05.122173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:05.122263 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:05.122579 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:05.122582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:05.122600 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:05.122711 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.122718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.123661 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.123665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.124691 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.124701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.125532 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.125536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.125636 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.125658 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:46:05.125764 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 14:46:05.125837 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":119,"thread_start_us":296,"threads_started":1}
I20260929 14:46:05.126394 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.121836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48575 (local address 127.0.98.1:43109)
0929 14:46:05.121961 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.121964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.122334 (+ 370us) server_negotiation.cc:407] Connection header received
0929 14:46:05.122406 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.122409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.122457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.122535 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:05.122885 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.123517 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.124837 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.125398 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.125491 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.125550 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.126167 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.126253 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:46:05.126497 18802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfc2f0edbe1447d8c0f3ac5a39dad8b" candidate_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" is_pre_election: true
I20260929 14:46:05.126871 18802 raft_consensus.cc:2454] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b50f5bc155473eb79bb3e8dba1573e in term 0.
I20260929 14:46:05.126899 18668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfc2f0edbe1447d8c0f3ac5a39dad8b" candidate_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" is_pre_election: true
I20260929 14:46:05.127173 18668 raft_consensus.cc:2454] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0b50f5bc155473eb79bb3e8dba1573e in term 0.
I20260929 14:46:05.127421 18872 leader_election.cc:302] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0b50f5bc155473eb79bb3e8dba1573e, ff731f0fb869419ebda9b4f11b24f9ab; no voters:
I20260929 14:46:05.127730 19007 raft_consensus.cc:2781] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:05.127811 19007 raft_consensus.cc:482] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:05.127842 19007 raft_consensus.cc:3037] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:05.129010 19007 raft_consensus.cc:504] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
I20260929 14:46:05.129375 19007 leader_election.cc:288] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [CANDIDATE]: Term 1 election: Requested vote from peers ff731f0fb869419ebda9b4f11b24f9ab (127.0.98.2:35909), 1b30d5cbf65746c8a2fba0d7f28ffe7e (127.0.98.1:43109)
I20260929 14:46:05.129740 18668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfc2f0edbe1447d8c0f3ac5a39dad8b" candidate_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e"
I20260929 14:46:05.129779 18802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfc2f0edbe1447d8c0f3ac5a39dad8b" candidate_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff731f0fb869419ebda9b4f11b24f9ab"
I20260929 14:46:05.129890 18802 raft_consensus.cc:3037] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:05.129904 18668 raft_consensus.cc:3037] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:05.131188 18668 raft_consensus.cc:2454] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b50f5bc155473eb79bb3e8dba1573e in term 1.
I20260929 14:46:05.131188 18802 raft_consensus.cc:2454] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0b50f5bc155473eb79bb3e8dba1573e in term 1.
I20260929 14:46:05.131640 18870 leader_election.cc:302] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b30d5cbf65746c8a2fba0d7f28ffe7e, b0b50f5bc155473eb79bb3e8dba1573e; no voters:
I20260929 14:46:05.132066 19007 raft_consensus.cc:2781] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:05.132325 19007 raft_consensus.cc:686] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 1 LEADER]: Becoming Leader. State: Replica: b0b50f5bc155473eb79bb3e8dba1573e, State: Running, Role: LEADER
I20260929 14:46:05.132702 19007 consensus_queue.cc:237] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } }
W20260929 14:46:05.134971 18849 tablet.cc:2369] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 14:46:05.135304 18715 tablet.cc:2369] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:05.136232 18522 catalog_manager.cc:5495] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e reported cstate change: term changed from 0 to 1, leader changed from <none> to b0b50f5bc155473eb79bb3e8dba1573e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b50f5bc155473eb79bb3e8dba1573e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 } health_report { overall_health: UNKNOWN } } }
I20260929 14:46:05.170163 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.166265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.0.98.3:34943)
0929 14:46:05.166433 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.166437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.166453 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:46:05.166527 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.166529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.166584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.166674 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:05.166937 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.167523 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.168415 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.168635 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.168726 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.169234 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:05.169669 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:05.169866 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.169987 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.170026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 14:46:05.176438 18802 raft_consensus.cc:1261] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 1 FOLLOWER]: Refusing update from remote peer b0b50f5bc155473eb79bb3e8dba1573e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:05.176585 18668 raft_consensus.cc:1261] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 1 FOLLOWER]: Refusing update from remote peer b0b50f5bc155473eb79bb3e8dba1573e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:05.177289 19009 consensus_queue.cc:1035] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b30d5cbf65746c8a2fba0d7f28ffe7e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:05.177517 19007 consensus_queue.cc:1035] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff731f0fb869419ebda9b4f11b24f9ab" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:05.184607 19012 mvcc.cc:205] Tried to move back new op lower bound from 7334679204554944512 to 7334679204386488320. Current Snapshot: MvccSnapshot[applied={T|T < 7334679204554944512}]
I20260929 14:46:05.188375 19013 mvcc.cc:205] Tried to move back new op lower bound from 7334679204554944512 to 7334679204386488320. Current Snapshot: MvccSnapshot[applied={T|T < 7334679204554944512}]
I20260929 14:46:05.189126 19015 mvcc.cc:205] Tried to move back new op lower bound from 7334679204554944512 to 7334679204386488320. Current Snapshot: MvccSnapshot[applied={T|T < 7334679204554944512 or (T in {7334679204554944512})}]
I20260929 14:46:05.194517 18522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:05.194687 18522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:05.197141 18522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10851c2788f8426e98471813fc1701e8: Sending DeleteTablet for 3 replicas of tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b
I20260929 14:46:05.198182 18916 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45752
I20260929 14:46:05.198150 18648 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49768
I20260929 14:46:05.198342 18782 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35566
I20260929 14:46:05.198863 19023 tablet_replica.cc:330] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: stopping tablet replica
I20260929 14:46:05.198865 19022 tablet_replica.cc:330] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e: stopping tablet replica
I20260929 14:46:05.199201 19023 raft_consensus.cc:2229] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:05.199210 19022 raft_consensus.cc:2229] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:05.199535 19023 raft_consensus.cc:2258] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:05.199535 19022 raft_consensus.cc:2258] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P 1b30d5cbf65746c8a2fba0d7f28ffe7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:05.200055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18585
I20260929 14:46:05.200717 19021 tablet_replica.cc:330] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: stopping tablet replica
I20260929 14:46:05.201068 19021 raft_consensus.cc:2229] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:46:05.201221 19023 ts_tablet_manager.cc:1905] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:05.201442 19021 raft_consensus.cc:2258] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:05.203887 19023 ts_tablet_manager.cc:1918] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:05.203980 19023 log.cc:1196] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/0bfc2f0edbe1447d8c0f3ac5a39dad8b
I20260929 14:46:05.204259 19023 ts_tablet_manager.cc:1939] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P ff731f0fb869419ebda9b4f11b24f9ab: Deleting consensus metadata
I20260929 14:46:05.204667 19021 ts_tablet_manager.cc:1905] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:05.205680 18510 catalog_manager.cc:4841] TS ff731f0fb869419ebda9b4f11b24f9ab (127.0.98.2:35909): tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b (table test-table [id=63af1207c38f45e2bdf54d3614b24dbe]) successfully deleted
I20260929 14:46:05.207005 19021 ts_tablet_manager.cc:1918] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:05.207088 19021 log.cc:1196] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/0bfc2f0edbe1447d8c0f3ac5a39dad8b
I20260929 14:46:05.207360 19021 ts_tablet_manager.cc:1939] T 0bfc2f0edbe1447d8c0f3ac5a39dad8b P b0b50f5bc155473eb79bb3e8dba1573e: Deleting consensus metadata
I20260929 14:46:05.208705 18507 catalog_manager.cc:4841] TS b0b50f5bc155473eb79bb3e8dba1573e (127.0.98.3:34943): tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b (table test-table [id=63af1207c38f45e2bdf54d3614b24dbe]) successfully deleted
W20260929 14:46:05.211519 18508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43109 (error 108)
I20260929 14:46:05.211709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18719
I20260929 14:46:05.214028 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.213712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43109 (local address 127.0.0.1:49778)
0929 14:46:05.213857 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:05.213935 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 14:46:05.214198 18508 catalog_manager.cc:4584] TS 1b30d5cbf65746c8a2fba0d7f28ffe7e (127.0.98.1:43109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bfc2f0edbe1447d8c0f3ac5a39dad8b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43109: connect: Connection refused (error 111)
I20260929 14:46:05.217912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18853
I20260929 14:46:05.222999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 18491
2026-09-29T14:46:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2438 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T14:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:05Z Disabled control of system clock
I20260929 14:46:05.252785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37203
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:05.365060 19028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:05.365368 19028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:05.365404 19028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:05.369876 19028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:05.369946 19028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:05.369961 19028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:05.369976 19028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:05.369990 19028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:05.375082 19028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:05.376219 19028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:05.377727 19028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:05.387074 19033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:05.387080 19036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:05.387080 19034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:05.387586 19028 server_base.cc:1034] running on GCE node
I20260929 14:46:05.388227 19028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:05.389528 19028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:05.390780 19028 hybrid_clock.cc:648] HybridClock initialized: now 1790693165390740 us; error 68 us; skew 500 ppm
I20260929 14:46:05.393237 19028 webserver.cc:460] Webserver started at http://127.0.98.62:37935/ using document root <none> and password file <none>
I20260929 14:46:05.393849 19028 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:05.393925 19028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:05.394129 19028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:05.396296 19028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "f2a1fa7b2be646278ac2b8632b69d8ca"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "0ef3dd4d4a108c3899f00a66a99de0e5"
server_key_iv: "98d1a17567b8ed2750c74b877ab9c580"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.396938 19028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2a1fa7b2be646278ac2b8632b69d8ca"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "0ef3dd4d4a108c3899f00a66a99de0e5"
server_key_iv: "98d1a17567b8ed2750c74b877ab9c580"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.401010 19028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:05.403672 19042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.405148 19028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:05.405287 19028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "f2a1fa7b2be646278ac2b8632b69d8ca"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "0ef3dd4d4a108c3899f00a66a99de0e5"
server_key_iv: "98d1a17567b8ed2750c74b877ab9c580"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.405400 19028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:05.427048 19028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:05.427811 19028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:05.428001 19028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:05.434916 19094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37203 every 8 connection(s)
I20260929 14:46:05.434911 19028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37203
I20260929 14:46:05.436023 19028 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:05.439350 19095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:05.439607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19028
I20260929 14:46:05.439730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:05.440037 392 external_mini_cluster.cc:1428] Setting key 24d9f767603aa612b3da204c83b7cacf
I20260929 14:46:05.445719 19095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: Bootstrap starting.
I20260929 14:46:05.448107 19095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:05.448941 19095 log.cc:824] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:05.449906 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.441683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.98.62:37203)
0929 14:46:05.442371 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.442386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.442424 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:46:05.443166 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.443186 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.443617 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.443952 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:05.444411 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.445423 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.446377 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.446865 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.447011 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.447892 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:05.447897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:05.447910 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:05.448334 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:05.448411 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:05.448435 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:05.448523 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.448672 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.448979 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":149,"threads_started":1}
I20260929 14:46:05.451193 19095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: No bootstrap required, opened a new log
I20260929 14:46:05.455210 19095 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260929 14:46:05.455520 19095 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:05.455587 19095 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a1fa7b2be646278ac2b8632b69d8ca, State: Initialized, Role: FOLLOWER
I20260929 14:46:05.456041 19095 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260929 14:46:05.456168 19095 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:05.456213 19095 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:05.456288 19095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:05.457281 19095 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260929 14:46:05.457635 19095 leader_election.cc:302] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a1fa7b2be646278ac2b8632b69d8ca; no voters:
I20260929 14:46:05.457929 19095 leader_election.cc:288] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:05.458060 19100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:05.458329 19100 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [term 1 LEADER]: Becoming Leader. State: Replica: f2a1fa7b2be646278ac2b8632b69d8ca, State: Running, Role: LEADER
I20260929 14:46:05.458628 19100 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } }
I20260929 14:46:05.459180 19095 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:05.460225 19102 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a1fa7b2be646278ac2b8632b69d8ca. Latest consensus state: current_term: 1 leader_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } } }
I20260929 14:46:05.460368 19102 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:05.460750 19109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:05.461169 19101 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a1fa7b2be646278ac2b8632b69d8ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37203 } } }
I20260929 14:46:05.461336 19101 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:05.463690 19109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:05.469481 19109 catalog_manager.cc:1347] Generated new cluster ID: 9c074ed1312f459dbd427b27cdbfad5d
I20260929 14:46:05.469570 19109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:05.479831 19109 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:05.481189 19109 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:05.488826 19109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: Generated new TSK 0
I20260929 14:46:05.489600 19109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:05.502923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:05.618489 19119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:05.618777 19119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:05.618815 19119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:05.622924 19119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:05.622995 19119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:05.623052 19119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:05.627916 19119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:05.629070 19119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:05.630483 19119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:05.641321 19127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:05.641448 19124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:05.641291 19125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:05.641691 19119 server_base.cc:1034] running on GCE node
I20260929 14:46:05.642148 19119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:05.642905 19119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:05.644084 19119 hybrid_clock.cc:648] HybridClock initialized: now 1790693165644046 us; error 57 us; skew 500 ppm
I20260929 14:46:05.653525 19119 webserver.cc:460] Webserver started at http://127.0.98.1:46825/ using document root <none> and password file <none>
I20260929 14:46:05.654174 19119 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:05.654258 19119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:05.654458 19119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:05.656610 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "3fb8f22c19f44e82a7284818cf9ca1d8"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "15eabe54a88db777064b1a722b452455"
server_key_iv: "4cf2e60c11a0a1cd912d8204ec9a08ec"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.657239 19119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3fb8f22c19f44e82a7284818cf9ca1d8"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "15eabe54a88db777064b1a722b452455"
server_key_iv: "4cf2e60c11a0a1cd912d8204ec9a08ec"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.661360 19119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:05.664106 19133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.665773 19119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:05.665910 19119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "3fb8f22c19f44e82a7284818cf9ca1d8"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "15eabe54a88db777064b1a722b452455"
server_key_iv: "4cf2e60c11a0a1cd912d8204ec9a08ec"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.666030 19119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:05.689389 19119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:05.690210 19119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:05.690409 19119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:05.691057 19119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:05.692137 19119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:05.692193 19119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.692250 19119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:05.692279 19119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.701889 19119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45873
I20260929 14:46:05.701929 19246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45873 every 8 connection(s)
I20260929 14:46:05.702880 19119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:05.710527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19119
I20260929 14:46:05.710664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:05.710974 392 external_mini_cluster.cc:1428] Setting key 3fc0947e82a79d5d2c613058016f0e7f
I20260929 14:46:05.713146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:05.713387 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.704830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37069 (local address 127.0.98.62:37203)
0929 14:46:05.705015 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.705019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.706286 (+ 1267us) server_negotiation.cc:407] Connection header received
0929 14:46:05.707765 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.707768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.707828 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.707904 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:05.709005 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.709578 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.711193 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.711451 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.711569 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.712890 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:05.712895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:05.712898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:05.712911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:05.712936 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:05.712946 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:05.713009 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.713164 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.713243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 14:46:05.714339 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.705315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37203 (local address 127.0.98.1:37069)
0929 14:46:05.706097 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:05.706153 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:05.707510 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:05.708087 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:05.708094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:05.708233 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:05.708816 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.708837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.709716 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.709719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.711002 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.711015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.711633 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.711636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.711835 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.712573 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:05.712575 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:05.712622 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:05.712705 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:05.713022 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:05.713033 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:05.713040 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:46:05.713155 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 14:46:05.713494 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":106,"threads_started":1}
I20260929 14:46:05.715624 19247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37203
I20260929 14:46:05.715917 19247 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:05.716610 19247 heartbeater.cc:511] Master 127.0.98.62:37203 requested a full tablet report, sending...
I20260929 14:46:05.718421 19059 ts_manager.cc:194] Registered new tserver with Master: 3fb8f22c19f44e82a7284818cf9ca1d8 (127.0.98.1:45873)
I20260929 14:46:05.719617 19059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37069
W20260929 14:46:05.825920 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:05.826201 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:05.826252 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:05.830404 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:05.830482 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:05.830562 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:05.835634 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:05.836748 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:05.838203 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:05.847973 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:05.848063 19250 server_base.cc:1034] running on GCE node
W20260929 14:46:05.847968 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:05.847968 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:05.848629 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:05.849318 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:05.850484 19250 hybrid_clock.cc:648] HybridClock initialized: now 1790693165850455 us; error 40 us; skew 500 ppm
I20260929 14:46:05.852455 19250 webserver.cc:460] Webserver started at http://127.0.98.2:36495/ using document root <none> and password file <none>
I20260929 14:46:05.853044 19250 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:05.853112 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:05.853278 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:05.855379 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "bffb988eb670416886a12d08409dab01"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "188e51f38d99fec8a54c388bc602e957"
server_key_iv: "69d222e84b56c5bfbdb21e32992e5685"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.855887 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bffb988eb670416886a12d08409dab01"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "188e51f38d99fec8a54c388bc602e957"
server_key_iv: "69d222e84b56c5bfbdb21e32992e5685"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.859601 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:05.862213 19264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.863607 19250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:46:05.863731 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "bffb988eb670416886a12d08409dab01"
format_stamp: "Formatted at 2026-09-29 14:46:05 on dist-test-slave-g8hp"
server_key: "188e51f38d99fec8a54c388bc602e957"
server_key_iv: "69d222e84b56c5bfbdb21e32992e5685"
server_key_version: "encryptionkey@0"
I20260929 14:46:05.863830 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:05.880235 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:05.880951 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:05.881100 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:05.881661 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:05.882591 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:05.882650 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.882691 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:05.882705 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:05.892200 19250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42427
I20260929 14:46:05.892248 19377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42427 every 8 connection(s)
I20260929 14:46:05.893239 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:46:05.900178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19250
I20260929 14:46:05.900357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:46:05.900637 392 external_mini_cluster.cc:1428] Setting key 32a47bd9a7b3d4e28f6612a1ec28c37d
I20260929 14:46:05.902842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--builtin_ntp_servers=127.0.98.20:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:05.903867 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.895371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59045 (local address 127.0.98.62:37203)
0929 14:46:05.895537 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:05.895541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:05.896755 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 14:46:05.898250 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:05.898253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:05.898306 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:05.898429 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:05.899630 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.900241 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.901695 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.901958 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.902055 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.903383 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:05.903387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:05.903390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:05.903404 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:05.903429 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:05.903438 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:05.903513 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:05.903645 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 14:46:05.903708 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 14:46:05.904942 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:05.895771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37203 (local address 127.0.98.2:59045)
0929 14:46:05.896501 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:05.896547 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:05.898012 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:05.898703 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:05.898715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:05.898837 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:05.899427 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.899442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.900389 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.900392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.901576 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:05.901589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:05.902146 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:05.902151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:05.902348 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:05.903081 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:05.903082 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:05.903133 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:05.903222 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:05.903517 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:05.903522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:05.903526 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 14:46:05.903635 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 14:46:05.903902 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":117,"threads_started":1}
I20260929 14:46:05.906268 19378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37203
I20260929 14:46:05.906555 19378 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:05.907267 19378 heartbeater.cc:511] Master 127.0.98.62:37203 requested a full tablet report, sending...
I20260929 14:46:05.908612 19059 ts_manager.cc:194] Registered new tserver with Master: bffb988eb670416886a12d08409dab01 (127.0.98.2:42427)
I20260929 14:46:05.909480 19059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59045
W20260929 14:46:06.017843 19381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:06.018119 19381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:06.018158 19381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:06.022351 19381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:06.022425 19381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:06.022481 19381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:06.027357 19381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:06.028450 19381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:06.029891 19381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:06.040030 19386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:06.040048 19389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:06.040030 19387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:06.040432 19381 server_base.cc:1034] running on GCE node
I20260929 14:46:06.040905 19381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:06.041543 19381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:06.042798 19381 hybrid_clock.cc:648] HybridClock initialized: now 1790693166042690 us; error 34 us; skew 500 ppm
I20260929 14:46:06.045207 19381 webserver.cc:460] Webserver started at http://127.0.98.3:35217/ using document root <none> and password file <none>
I20260929 14:46:06.045754 19381 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:06.045814 19381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:06.045975 19381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:06.048029 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03"
format_stamp: "Formatted at 2026-09-29 14:46:06 on dist-test-slave-g8hp"
server_key: "a6b7449ccf42341b55c68eb456d31907"
server_key_iv: "699fb5654fb932102dac54c512750623"
server_key_version: "encryptionkey@0"
I20260929 14:46:06.048555 19381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03"
format_stamp: "Formatted at 2026-09-29 14:46:06 on dist-test-slave-g8hp"
server_key: "a6b7449ccf42341b55c68eb456d31907"
server_key_iv: "699fb5654fb932102dac54c512750623"
server_key_version: "encryptionkey@0"
I20260929 14:46:06.052491 19381 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:06.055125 19395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:06.056526 19381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 14:46:06.056746 19381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03"
format_stamp: "Formatted at 2026-09-29 14:46:06 on dist-test-slave-g8hp"
server_key: "a6b7449ccf42341b55c68eb456d31907"
server_key_iv: "699fb5654fb932102dac54c512750623"
server_key_version: "encryptionkey@0"
I20260929 14:46:06.056854 19381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:06.086593 19381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:06.087301 19381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:06.087469 19381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:06.088032 19381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:06.089035 19381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:06.089084 19381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:06.089126 19381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:06.089147 19381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:06.098536 19381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36317
I20260929 14:46:06.098577 19508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36317 every 8 connection(s)
I20260929 14:46:06.099538 19381 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:06.100476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19381
I20260929 14:46:06.100589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:06.100924 392 external_mini_cluster.cc:1428] Setting key 8c9d6eb6e5681e317feca49e7cf9332d
I20260929 14:46:06.110324 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.101988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49117 (local address 127.0.98.62:37203)
0929 14:46:06.102129 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.102134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.103439 (+ 1305us) server_negotiation.cc:407] Connection header received
0929 14:46:06.104974 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.104977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.105036 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.105112 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:06.106167 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.106814 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.108157 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.108412 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.108511 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.109815 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:06.109818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:06.109820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:06.109834 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:06.109855 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:06.109863 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:06.109926 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.110098 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.110158 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:46:06.111228 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.102376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37203 (local address 127.0.98.3:49117)
0929 14:46:06.103292 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.103353 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.104708 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.105345 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.105353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.105449 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:06.105994 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.106013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.106970 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.106973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.108020 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.108030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.108616 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.108621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.108798 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.109507 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:06.109509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:06.109553 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:06.109627 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:06.109948 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:06.109957 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:06.109963 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.110076 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.110331 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":159,"threads_started":1}
I20260929 14:46:06.112463 19509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37203
I20260929 14:46:06.112684 19509 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:06.113261 19509 heartbeater.cc:511] Master 127.0.98.62:37203 requested a full tablet report, sending...
I20260929 14:46:06.114391 19059 ts_manager.cc:194] Registered new tserver with Master: b2e7ce8b9d934c9c8e95d7c9f3885c03 (127.0.98.3:36317)
I20260929 14:46:06.115036 19059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49117
I20260929 14:46:06.115638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:06.122430 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.117559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.62:37203)
0929 14:46:06.117796 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.117800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.117813 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:06.117907 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.117910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.117949 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.118015 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:06.118362 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.118984 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.120044 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.120328 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.120430 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.121596 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:06.121600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:06.121603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:06.121629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:06.121654 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:06.121665 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:06.121943 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.122153 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.122231 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:46:06.126231 19058 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:06.129323 19058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:06.145283 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.138903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45873 (local address 127.0.0.1:33278)
0929 14:46:06.139325 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.139378 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.139502 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.140637 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.140643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.140679 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:06.140991 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.141001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.142449 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.142452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.143527 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.143534 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144576 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.144728 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.144825 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.144927 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.145076 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":110,"spinlock_wait_cycles":6016,"thread_start_us":87,"threads_started":1}
I20260929 14:46:06.145496 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.139155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.0.98.1:45873)
0929 14:46:06.139590 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.139605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.139633 (+ 28us) server_negotiation.cc:407] Connection header received
0929 14:46:06.139717 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.139723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.140496 (+ 773us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.140749 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:06.141147 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.142322 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.143655 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144445 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144804 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.145163 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.145216 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.145322 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":103,"threads_started":1}
I20260929 14:46:06.145573 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.138914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.0.98.2:42427)
0929 14:46:06.139408 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.139420 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.139816 (+ 396us) server_negotiation.cc:407] Connection header received
0929 14:46:06.140037 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.140043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.140269 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.140419 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:06.141122 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.141935 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.143479 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144366 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144482 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.144745 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.145232 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.145325 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":161,"threads_started":1}
I20260929 14:46:06.145900 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.139765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36317 (local address 127.0.0.1:33614)
0929 14:46:06.140060 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.140072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.140152 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.141098 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.141101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.141119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:06.141189 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.141194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.142488 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.142491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.143577 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.143587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144786 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.144958 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.144981 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.145044 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.145101 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":28,"thread_start_us":97,"threads_started":1}
I20260929 14:46:06.146037 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.138903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42427 (local address 127.0.0.1:54872)
0929 14:46:06.139712 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.139724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.139888 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.140637 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.140643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.140678 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:06.140987 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.140994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.142054 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.142058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.143346 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.143357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144508 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.144724 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.144799 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.145779 (+ 980us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.145878 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":263,"spinlock_wait_cycles":6272,"thread_start_us":453,"threads_started":1}
I20260929 14:46:06.146242 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.140042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.98.3:36317)
0929 14:46:06.140485 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.140496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.140518 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:46:06.140588 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.140592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.140771 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.140974 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:06.141307 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.142404 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.143710 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.144436 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.144518 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.144698 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.145217 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.145325 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":150,"threads_started":1}
I20260929 14:46:06.148685 19312 tablet_service.cc:1467] Processing CreateTablet for tablet 56e6d71220404804b36d42881cb1d984 (DEFAULT_TABLE table=test-table [id=27a5500566ab4418a080a90ff4b8c6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:06.148790 19442 tablet_service.cc:1467] Processing CreateTablet for tablet 56e6d71220404804b36d42881cb1d984 (DEFAULT_TABLE table=test-table [id=27a5500566ab4418a080a90ff4b8c6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:06.148792 19181 tablet_service.cc:1467] Processing CreateTablet for tablet 56e6d71220404804b36d42881cb1d984 (DEFAULT_TABLE table=test-table [id=27a5500566ab4418a080a90ff4b8c6ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:06.149796 19312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e6d71220404804b36d42881cb1d984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:06.149945 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e6d71220404804b36d42881cb1d984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:06.150049 19181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e6d71220404804b36d42881cb1d984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:06.156116 19526 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Bootstrap starting.
I20260929 14:46:06.158257 19527 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Bootstrap starting.
I20260929 14:46:06.158668 19528 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Bootstrap starting.
I20260929 14:46:06.159087 19526 tablet_bootstrap.cc:654] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:06.160003 19526 log.cc:824] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:06.160385 19527 tablet_bootstrap.cc:654] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:06.160727 19528 tablet_bootstrap.cc:654] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:06.161314 19527 log.cc:824] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:06.161499 19528 log.cc:824] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:06.162638 19526 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: No bootstrap required, opened a new log
I20260929 14:46:06.162868 19526 ts_tablet_manager.cc:1397] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:46:06.163555 19527 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: No bootstrap required, opened a new log
I20260929 14:46:06.163741 19527 ts_tablet_manager.cc:1397] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:46:06.166493 19528 tablet_bootstrap.cc:492] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: No bootstrap required, opened a new log
I20260929 14:46:06.166709 19528 ts_tablet_manager.cc:1397] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260929 14:46:06.166975 19527 raft_consensus.cc:348] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.166988 19526 raft_consensus.cc:348] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.167233 19527 raft_consensus.cc:374] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:06.167253 19526 raft_consensus.cc:374] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:06.167289 19527 raft_consensus.cc:729] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb8f22c19f44e82a7284818cf9ca1d8, State: Initialized, Role: FOLLOWER
I20260929 14:46:06.167300 19526 raft_consensus.cc:729] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffb988eb670416886a12d08409dab01, State: Initialized, Role: FOLLOWER
I20260929 14:46:06.167754 19527 consensus_queue.cc:260] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.167765 19526 consensus_queue.cc:260] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.168444 19247 heartbeater.cc:503] Master 127.0.98.62:37203 was elected leader, sending a full tablet report...
I20260929 14:46:06.168941 19526 ts_tablet_manager.cc:1428] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:46:06.169292 19378 heartbeater.cc:503] Master 127.0.98.62:37203 was elected leader, sending a full tablet report...
I20260929 14:46:06.170066 19527 ts_tablet_manager.cc:1428] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260929 14:46:06.172075 19528 raft_consensus.cc:348] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.172396 19528 raft_consensus.cc:374] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:06.172457 19528 raft_consensus.cc:729] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e7ce8b9d934c9c8e95d7c9f3885c03, State: Initialized, Role: FOLLOWER
I20260929 14:46:06.172988 19528 consensus_queue.cc:260] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.173700 19509 heartbeater.cc:503] Master 127.0.98.62:37203 was elected leader, sending a full tablet report...
I20260929 14:46:06.174162 19528 ts_tablet_manager.cc:1428] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260929 14:46:06.204181 19248 tablet.cc:2369] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:06.325210 19532 raft_consensus.cc:482] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:06.325443 19532 raft_consensus.cc:504] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.326714 19532 leader_election.cc:288] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffb988eb670416886a12d08409dab01 (127.0.98.2:42427), b2e7ce8b9d934c9c8e95d7c9f3885c03 (127.0.98.3:36317)
I20260929 14:46:06.330873 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.327086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56693 (local address 127.0.98.2:42427)
0929 14:46:06.327218 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.327223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.327470 (+ 247us) server_negotiation.cc:407] Connection header received
0929 14:46:06.327576 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.327579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.327642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.327713 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:06.328044 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.328719 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.329880 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.330276 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.330367 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.330430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.330670 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.330742 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:46:06.330884 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.327224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42533 (local address 127.0.98.3:36317)
0929 14:46:06.327348 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.327352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.327581 (+ 229us) server_negotiation.cc:407] Connection header received
0929 14:46:06.327768 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.327771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.327817 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.327898 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:06.328178 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.328681 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.329883 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.330283 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.330377 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.330433 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.330700 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.330778 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:46:06.330894 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.326995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36317 (local address 127.0.98.1:42533)
0929 14:46:06.327466 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.327480 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.327593 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.327922 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.327924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.327938 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:06.328042 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.328049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.328811 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.328814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.329747 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.329756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.330421 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.330424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.330556 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.330591 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.330662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.330741 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":93,"thread_start_us":95,"threads_started":1}
I20260929 14:46:06.330899 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.326995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42427 (local address 127.0.98.1:56693)
0929 14:46:06.327359 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.327372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:06.327487 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:06.327752 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:06.327756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:06.327774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:06.327921 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.327929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.328851 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.328853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.329750 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:06.329759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.330408 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.330412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:06.330545 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.330572 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 14:46:06.330630 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 14:46:06.330725 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":48,"thread_start_us":102,"threads_started":1}
I20260929 14:46:06.331562 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e6d71220404804b36d42881cb1d984" candidate_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" is_pre_election: true
I20260929 14:46:06.331594 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e6d71220404804b36d42881cb1d984" candidate_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb988eb670416886a12d08409dab01" is_pre_election: true
I20260929 14:46:06.331872 19463 raft_consensus.cc:2454] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb8f22c19f44e82a7284818cf9ca1d8 in term 0.
I20260929 14:46:06.331871 19332 raft_consensus.cc:2454] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb8f22c19f44e82a7284818cf9ca1d8 in term 0.
I20260929 14:46:06.332379 19137 leader_election.cc:302] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb8f22c19f44e82a7284818cf9ca1d8, bffb988eb670416886a12d08409dab01; no voters:
I20260929 14:46:06.332674 19532 raft_consensus.cc:2781] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:06.332767 19532 raft_consensus.cc:482] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:06.332809 19532 raft_consensus.cc:3037] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:06.333880 19532 raft_consensus.cc:504] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.334235 19532 leader_election.cc:288] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [CANDIDATE]: Term 1 election: Requested vote from peers bffb988eb670416886a12d08409dab01 (127.0.98.2:42427), b2e7ce8b9d934c9c8e95d7c9f3885c03 (127.0.98.3:36317)
I20260929 14:46:06.334622 19332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e6d71220404804b36d42881cb1d984" candidate_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffb988eb670416886a12d08409dab01"
I20260929 14:46:06.334657 19463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e6d71220404804b36d42881cb1d984" candidate_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03"
I20260929 14:46:06.334735 19332 raft_consensus.cc:3037] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:06.334826 19463 raft_consensus.cc:3037] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:06.335820 19332 raft_consensus.cc:2454] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb8f22c19f44e82a7284818cf9ca1d8 in term 1.
I20260929 14:46:06.335973 19463 raft_consensus.cc:2454] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb8f22c19f44e82a7284818cf9ca1d8 in term 1.
I20260929 14:46:06.336163 19137 leader_election.cc:302] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb8f22c19f44e82a7284818cf9ca1d8, bffb988eb670416886a12d08409dab01; no voters:
I20260929 14:46:06.336333 19532 raft_consensus.cc:2781] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:06.336549 19532 raft_consensus.cc:686] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb8f22c19f44e82a7284818cf9ca1d8, State: Running, Role: LEADER
I20260929 14:46:06.336954 19532 consensus_queue.cc:237] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } }
I20260929 14:46:06.340337 19059 catalog_manager.cc:5495] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb8f22c19f44e82a7284818cf9ca1d8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fb8f22c19f44e82a7284818cf9ca1d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 } health_report { overall_health: UNKNOWN } } }
W20260929 14:46:06.350678 19510 tablet.cc:2369] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:06.393822 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.389585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.1:45873)
0929 14:46:06.389731 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:06.389735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:06.389749 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:46:06.389820 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:06.389823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:06.389880 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:06.389986 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:06.390296 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.390929 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.391923 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:06.392152 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:06.392245 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:06.392724 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:06.393263 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:06.393386 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:06.393667 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 14:46:06.393708 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 14:46:06.394366 19379 tablet.cc:2369] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:06.400262 19463 raft_consensus.cc:1261] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 1 FOLLOWER]: Refusing update from remote peer 3fb8f22c19f44e82a7284818cf9ca1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:06.400308 19332 raft_consensus.cc:1261] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 1 FOLLOWER]: Refusing update from remote peer 3fb8f22c19f44e82a7284818cf9ca1d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:06.401119 19532 consensus_queue.cc:1035] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2e7ce8b9d934c9c8e95d7c9f3885c03" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:06.401396 19537 consensus_queue.cc:1035] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffb988eb670416886a12d08409dab01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:06.411283 19542 mvcc.cc:205] Tried to move back new op lower bound from 7334679209567596544 to 7334679209319485440. Current Snapshot: MvccSnapshot[applied={T|T < 7334679209567596544}]
I20260929 14:46:06.420468 19057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:06.420651 19057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:06.423254 19057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2a1fa7b2be646278ac2b8632b69d8ca: Sending DeleteTablet for 3 replicas of tablet 56e6d71220404804b36d42881cb1d984
I20260929 14:46:06.424139 19442 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e6d71220404804b36d42881cb1d984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:06 UTC) from {username='slave'} at 127.0.0.1:33614
I20260929 14:46:06.424162 19181 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e6d71220404804b36d42881cb1d984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:06 UTC) from {username='slave'} at 127.0.0.1:33278
I20260929 14:46:06.424659 19550 tablet_replica.cc:330] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: stopping tablet replica
I20260929 14:46:06.424690 19312 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e6d71220404804b36d42881cb1d984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:06 UTC) from {username='slave'} at 127.0.0.1:54872
I20260929 14:46:06.425057 19551 tablet_replica.cc:330] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8: stopping tablet replica
I20260929 14:46:06.425192 19552 tablet_replica.cc:330] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: stopping tablet replica
I20260929 14:46:06.425302 19550 raft_consensus.cc:2229] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:06.425370 19551 raft_consensus.cc:2229] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:46:06.425432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19119
I20260929 14:46:06.425787 19551 raft_consensus.cc:2258] T 56e6d71220404804b36d42881cb1d984 P 3fb8f22c19f44e82a7284818cf9ca1d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:06.425467 19552 raft_consensus.cc:2229] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:06.426378 19552 raft_consensus.cc:2258] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:06.426775 19550 raft_consensus.cc:2258] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:06.428303 19552 ts_tablet_manager.cc:1905] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:06.428309 19550 ts_tablet_manager.cc:1905] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:06.431417 19550 ts_tablet_manager.cc:1918] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:06.431417 19552 ts_tablet_manager.cc:1918] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:06.431558 19552 log.cc:1196] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/56e6d71220404804b36d42881cb1d984
I20260929 14:46:06.431558 19550 log.cc:1196] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/56e6d71220404804b36d42881cb1d984
I20260929 14:46:06.431890 19550 ts_tablet_manager.cc:1939] T 56e6d71220404804b36d42881cb1d984 P b2e7ce8b9d934c9c8e95d7c9f3885c03: Deleting consensus metadata
I20260929 14:46:06.431890 19552 ts_tablet_manager.cc:1939] T 56e6d71220404804b36d42881cb1d984 P bffb988eb670416886a12d08409dab01: Deleting consensus metadata
W20260929 14:46:06.433005 19045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45873 (error 108)
I20260929 14:46:06.433207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19250
I20260929 14:46:06.433652 19045 catalog_manager.cc:4841] TS b2e7ce8b9d934c9c8e95d7c9f3885c03 (127.0.98.3:36317): tablet 56e6d71220404804b36d42881cb1d984 (table test-table [id=27a5500566ab4418a080a90ff4b8c6ce]) successfully deleted
I20260929 14:46:06.435323 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.434992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45873 (local address 127.0.0.1:33294)
0929 14:46:06.435150 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.435237 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 14:46:06.435581 19045 catalog_manager.cc:4584] TS 3fb8f22c19f44e82a7284818cf9ca1d8 (127.0.98.1:45873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56e6d71220404804b36d42881cb1d984: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45873: connect: Connection refused (error 111)
I20260929 14:46:06.438598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19381
I20260929 14:46:06.439077 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:06.438769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42427 (local address 127.0.0.1:54888)
0929 14:46:06.438950 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:06.438998 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 14:46:06.446182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19028
2026-09-29T14:46:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1222 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:06 dist-test-slave-g8hp krb5kdc[19556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:06 dist-test-slave-g8hp krb5kdc[19556](info): set up 2 sockets
Sep 29 14:46:06 dist-test-slave-g8hp krb5kdc[19556](info): commencing operation
krb5kdc: starting...
W20260929 14:46:07.424958 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:07 dist-test-slave-g8hp krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:07.557263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:07.671267 19569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:07.671597 19569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:07.671656 19569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:07.676216 19569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:07.676304 19569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:07.676330 19569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:07.676362 19569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:07.676383 19569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:07.682502 19569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:07.683717 19569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:07.684782 19569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:07.693955 19574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:07.694079 19569 server_base.cc:1034] running on GCE node
W20260929 14:46:07.693928 19575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:07.693928 19577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:07.694892 19569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:07.696101 19569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:07.697270 19569 hybrid_clock.cc:648] HybridClock initialized: now 1790693167697249 us; error 47 us; skew 500 ppm
Sep 29 14:46:07 dist-test-slave-g8hp krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:07.700446 19569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:07.701546 19569 webserver.cc:460] Webserver started at http://127.0.98.62:35129/ using document root <none> and password file <none>
I20260929 14:46:07.702129 19569 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:07.702206 19569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:07.702409 19569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:07.704630 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "a1f9700d04df4363a5fa9a2bbe5bf643"
format_stamp: "Formatted at 2026-09-29 14:46:07 on dist-test-slave-g8hp"
server_key: "deff49cc8f1e5fe41e95fa02fd55a982"
server_key_iv: "abfee206146062397c45dfff200d5dc8"
server_key_version: "encryptionkey@0"
I20260929 14:46:07.705281 19569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "a1f9700d04df4363a5fa9a2bbe5bf643"
format_stamp: "Formatted at 2026-09-29 14:46:07 on dist-test-slave-g8hp"
server_key: "deff49cc8f1e5fe41e95fa02fd55a982"
server_key_iv: "abfee206146062397c45dfff200d5dc8"
server_key_version: "encryptionkey@0"
I20260929 14:46:07.709337 19569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:07.712208 19584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:07.713718 19569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:07.713857 19569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "a1f9700d04df4363a5fa9a2bbe5bf643"
format_stamp: "Formatted at 2026-09-29 14:46:07 on dist-test-slave-g8hp"
server_key: "deff49cc8f1e5fe41e95fa02fd55a982"
server_key_iv: "abfee206146062397c45dfff200d5dc8"
server_key_version: "encryptionkey@0"
I20260929 14:46:07.713979 19569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:07.741086 19569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:07.744805 19569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:07.745042 19569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:07.752699 19569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41293
I20260929 14:46:07.752794 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41293 every 8 connection(s)
I20260929 14:46:07.753849 19569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:07.754038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19569
I20260929 14:46:07.754181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:07.754532 392 external_mini_cluster.cc:1428] Setting key f4d563e6a53475ce34bfd028d77f83a8
I20260929 14:46:07.758368 19637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 14:46:07 dist-test-slave-g8hp krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:07.764886 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: Bootstrap starting.
I20260929 14:46:07.767930 19637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:07.768930 19637 log.cc:824] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:07.770848 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:07.755973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.0.98.62:41293)
0929 14:46:07.756398 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:07.756417 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:07.756455 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:46:07.757261 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:07.757280 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:07.757740 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:07.758154 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:07.759077 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:07.760090 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:07.761006 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:07.761483 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:07.761641 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:07.765001 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:07.765008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:07.765027 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:07.765178 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:07.767329 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:07.767835 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:07.767843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:07.767850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:07.768033 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:07.768342 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:07.768345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:07.768347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:07.768948 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:07.769163 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:07.769528 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 14:46:07.769855 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260929 14:46:07.771399 19637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: No bootstrap required, opened a new log
I20260929 14:46:07.775298 19637 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260929 14:46:07.775578 19637 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:07.775616 19637 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f9700d04df4363a5fa9a2bbe5bf643, State: Initialized, Role: FOLLOWER
I20260929 14:46:07.776018 19637 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260929 14:46:07.776137 19637 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:07.776185 19637 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:07.776259 19637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:07.777292 19637 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260929 14:46:07.777839 19637 leader_election.cc:302] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1f9700d04df4363a5fa9a2bbe5bf643; no voters:
I20260929 14:46:07.778330 19637 leader_election.cc:288] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:07.778439 19642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:07.778729 19642 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [term 1 LEADER]: Becoming Leader. State: Replica: a1f9700d04df4363a5fa9a2bbe5bf643, State: Running, Role: LEADER
I20260929 14:46:07.779049 19642 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } }
I20260929 14:46:07.779652 19637 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:07.780755 19644 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f9700d04df4363a5fa9a2bbe5bf643. Latest consensus state: current_term: 1 leader_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } } }
I20260929 14:46:07.780901 19644 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:07.781204 19643 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1f9700d04df4363a5fa9a2bbe5bf643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41293 } } }
I20260929 14:46:07.781296 19643 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:07.781412 19651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:07.785163 19651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:07.790802 19651 catalog_manager.cc:1347] Generated new cluster ID: 64ca0e9271a848af989a1c62d347093b
I20260929 14:46:07.790904 19651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:07.805632 19651 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:07.806644 19651 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:07.821363 19651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: Generated new TSK 0
I20260929 14:46:07.822292 19651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 14:46:07.860801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:07.975607 19663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:07.975894 19663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:07.975939 19663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:07.980186 19663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:07.980258 19663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:07.980331 19663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:07.986409 19663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:07.987555 19663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:07.988633 19663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:07.999605 19668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:07.999773 19669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:07.999607 19671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:07.999871 19663 server_base.cc:1034] running on GCE node
I20260929 14:46:08.000458 19663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:08.001296 19663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:08.002507 19663 hybrid_clock.cc:648] HybridClock initialized: now 1790693168002473 us; error 46 us; skew 500 ppm
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:08.005964 19663 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 14:46:08.006878 19663 webserver.cc:460] Webserver started at http://127.0.98.1:36629/ using document root <none> and password file <none>
I20260929 14:46:08.007406 19663 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:08.007465 19663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:08.007637 19663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:08.009791 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "4f58a6203861417abc05101ca0114750"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "fec4ce3a28dbe25b1409d21544c83627"
server_key_iv: "42c5b4fbf00a9c9d350c42ece76411f9"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.010313 19663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4f58a6203861417abc05101ca0114750"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "fec4ce3a28dbe25b1409d21544c83627"
server_key_iv: "42c5b4fbf00a9c9d350c42ece76411f9"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.014372 19663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:08.017699 19678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.019339 19663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 14:46:08.019474 19663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "4f58a6203861417abc05101ca0114750"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "fec4ce3a28dbe25b1409d21544c83627"
server_key_iv: "42c5b4fbf00a9c9d350c42ece76411f9"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.019594 19663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:08.058147 19663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:08.061941 19663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:08.062107 19663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:08.062664 19663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:08.063527 19663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:08.063575 19663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.063616 19663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:08.063643 19663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.072597 19663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33051
I20260929 14:46:08.072791 19791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33051 every 8 connection(s)
I20260929 14:46:08.073617 19663 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:08.078163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19663
I20260929 14:46:08.078307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:08.078646 392 external_mini_cluster.cc:1428] Setting key d4eee41002f1c8713e23f83f6ee21c0d
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 14:46:08.089462 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.075789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59467 (local address 127.0.98.62:41293)
0929 14:46:08.075938 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.075942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.077076 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 14:46:08.078491 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.078494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.078548 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.078630 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:08.080227 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.080770 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.082349 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.082594 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.082695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.085443 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:08.085446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:08.085448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:08.085475 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:08.087032 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.087736 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.087739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.087740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.087818 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.088234 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.088237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.088238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.088435 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:08.088513 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.089172 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.089302 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:46:08.090330 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.076151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.1:59467)
0929 14:46:08.076807 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.076886 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.078219 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.078871 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.078881 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.079431 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:08.080046 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.080068 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.080939 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.080942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.082163 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.082223 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.082768 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.082773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.082958 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.083600 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:08.083617 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:08.085228 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:08.087170 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.087178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.087199 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.087578 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.087904 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.087906 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.087909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.088116 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.088598 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:08.088604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:08.088901 (+ 297us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.089093 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.089428 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 14:46:08.091830 19792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260929 14:46:08.092167 19792 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:08.092775 19792 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260929 14:46:08.094806 19601 ts_manager.cc:194] Registered new tserver with Master: 4f58a6203861417abc05101ca0114750 (127.0.98.1:33051)
I20260929 14:46:08.096423 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 14:46:08.114573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:08.231014 19797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:08.231411 19797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:08.231626 19797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:08.235890 19797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:08.235986 19797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:08.236074 19797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:08.241734 19797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:08.242841 19797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:08.243884 19797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:08.254024 19803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:08.254016 19805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:08.254031 19802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:08.254284 19797 server_base.cc:1034] running on GCE node
I20260929 14:46:08.254856 19797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:08.255594 19797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:08.256781 19797 hybrid_clock.cc:648] HybridClock initialized: now 1790693168256752 us; error 39 us; skew 500 ppm
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:08.260033 19797 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 14:46:08.261229 19797 webserver.cc:460] Webserver started at http://127.0.98.2:45691/ using document root <none> and password file <none>
I20260929 14:46:08.261827 19797 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:08.261904 19797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:08.262125 19797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:08.264348 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "904584b9309546dba1e49d9985874e67"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "76029f41986f1f14ba9541da383a2136"
server_key_iv: "1cf271c24cfbc767c0fe2094b419a4b1"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.265007 19797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "904584b9309546dba1e49d9985874e67"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "76029f41986f1f14ba9541da383a2136"
server_key_iv: "1cf271c24cfbc767c0fe2094b419a4b1"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.269253 19797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:08.272432 19812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.273926 19797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:08.274076 19797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "904584b9309546dba1e49d9985874e67"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "76029f41986f1f14ba9541da383a2136"
server_key_iv: "1cf271c24cfbc767c0fe2094b419a4b1"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.274201 19797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:08.287137 19797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:08.290495 19797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:08.290690 19797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:08.291304 19797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:08.292162 19797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:08.292209 19797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.292263 19797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:08.292290 19797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.301017 19797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39645
I20260929 14:46:08.301072 19925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39645 every 8 connection(s)
I20260929 14:46:08.301973 19797 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:08.311929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19797
I20260929 14:46:08.312064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:46:08.312357 392 external_mini_cluster.cc:1428] Setting key 5c28b56bb245353e90bf6bf012100b1c
I20260929 14:46:08.316107 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.303891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34825 (local address 127.0.98.62:41293)
0929 14:46:08.304028 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.304032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.304946 (+ 914us) server_negotiation.cc:407] Connection header received
0929 14:46:08.306318 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.306321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.306385 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.306457 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:08.307751 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.308280 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.309498 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.309721 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.309810 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.312541 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:08.312544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:08.312546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:08.312574 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:08.314044 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.314566 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.314570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.314572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.314652 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.314999 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.315003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.315005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.315188 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:08.315276 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.315810 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.315941 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:46:08.317255 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.304212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.2:34825)
0929 14:46:08.304758 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.304804 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.306080 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.306660 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.306668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.307060 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:08.307576 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.307591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.308413 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.308416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.309364 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.309375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.309875 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.309879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.310041 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.310616 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:08.310633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:08.312329 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:08.314166 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.314173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.314191 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.314458 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.314743 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.314746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.314748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.314900 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.315297 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:08.315304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:08.315568 (+ 264us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.315751 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.316127 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":125,"threads_started":1}
I20260929 14:46:08.318487 19926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260929 14:46:08.318809 19926 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:08.319567 19926 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260929 14:46:08.320819 19601 ts_manager.cc:194] Registered new tserver with Master: 904584b9309546dba1e49d9985874e67 (127.0.98.2:39645)
I20260929 14:46:08.321447 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34825
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 14:46:08.343967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:08.458231 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:08.458518 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:08.458561 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:08.462913 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:08.462987 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:08.463048 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:08.468565 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:08.469731 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:08.470733 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:08.481142 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:08.481142 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:08.481177 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:08.481323 19931 server_base.cc:1034] running on GCE node
I20260929 14:46:08.481853 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:08.482527 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:08.483700 19931 hybrid_clock.cc:648] HybridClock initialized: now 1790693168483674 us; error 38 us; skew 500 ppm
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:08.486636 19931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 14:46:08.487556 19931 webserver.cc:460] Webserver started at http://127.0.98.3:36853/ using document root <none> and password file <none>
I20260929 14:46:08.488152 19931 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:08.488222 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:08.488425 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:08.490672 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "00473f8181b046db8c2031b7fe8463ca"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "6e5cf4ab7412c2015f9087d3cd7fcf93"
server_key_iv: "1796d42573ea9201dba3a3993250cdf8"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.491225 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "00473f8181b046db8c2031b7fe8463ca"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "6e5cf4ab7412c2015f9087d3cd7fcf93"
server_key_iv: "1796d42573ea9201dba3a3993250cdf8"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.495137 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:08.497876 19946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.499274 19931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:08.499415 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "00473f8181b046db8c2031b7fe8463ca"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "6e5cf4ab7412c2015f9087d3cd7fcf93"
server_key_iv: "1796d42573ea9201dba3a3993250cdf8"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.499559 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:08.517481 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:08.521005 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:08.521186 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:08.521803 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:08.522711 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:08.522760 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.522809 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:08.522836 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.532122 19931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33197
I20260929 14:46:08.532189 20059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33197 every 8 connection(s)
I20260929 14:46:08.533186 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:08.541091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 19931
I20260929 14:46:08.541224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:08.541637 392 external_mini_cluster.cc:1428] Setting key 4476de815e38e82b75baadf9e755e5b9
Sep 29 14:46:08 dist-test-slave-g8hp krb5kdc[19556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:08.547924 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.535323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38317 (local address 127.0.98.62:41293)
0929 14:46:08.535517 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.535522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.536328 (+ 806us) server_negotiation.cc:407] Connection header received
0929 14:46:08.537765 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.537768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.537829 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.537909 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:08.539233 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.539884 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.541215 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.541540 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.541665 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.544352 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:08.544355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:08.544357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:08.544385 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:08.545827 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.546337 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.546340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.546342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.546424 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.546800 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.546802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.546803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.547030 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:08.547107 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.547681 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.547798 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 14:46:08.548952 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.535688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41293 (local address 127.0.98.3:38317)
0929 14:46:08.536163 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.536209 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.537464 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.538094 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.538103 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.538493 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:08.539075 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.539089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.540049 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.540053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.541077 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.541088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.541746 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.541751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.541940 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.542520 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:08.542537 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:08.544164 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:08.545944 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.545949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.545964 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.546245 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.546495 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:08.546497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:08.546499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:08.546718 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:08.547151 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:08.547156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:08.547422 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.547615 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.547908 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
I20260929 14:46:08.550186 20060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41293
I20260929 14:46:08.550424 20060 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:08.550907 20060 heartbeater.cc:511] Master 127.0.98.62:41293 requested a full tablet report, sending...
I20260929 14:46:08.552127 19601 ts_manager.cc:194] Registered new tserver with Master: 00473f8181b046db8c2031b7fe8463ca (127.0.98.3:33197)
I20260929 14:46:08.552707 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38317
I20260929 14:46:08.555780 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:08.565852 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.557838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.0.98.62:41293)
0929 14:46:08.557986 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.557990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.558088 (+ 98us) server_negotiation.cc:407] Connection header received
0929 14:46:08.558229 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.558232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.558278 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.558348 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:08.559203 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.559749 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.560635 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.560895 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.561001 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.562614 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:08.562617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:08.562620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:08.562647 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:08.564125 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.564574 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.564577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.564578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.564682 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:08.564933 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:08.564936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:08.564938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:08.565198 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:08.565307 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.565512 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.565682 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 14:46:08.569114 19601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:08.572217 19601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:08.587061 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.581434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33197 (local address 127.0.0.1:60394)
0929 14:46:08.581834 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.581891 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.582021 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.582734 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.582740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.582765 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:08.582911 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.582919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.584308 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.584310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.585277 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.585285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.586507 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.586510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.586626 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.586667 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.586783 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.586905 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
I20260929 14:46:08.587106 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.581602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.3:33197)
0929 14:46:08.581979 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.581991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.582015 (+ 24us) server_negotiation.cc:407] Connection header received
0929 14:46:08.582156 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.582162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.582372 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.582525 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:08.583019 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.584169 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.585403 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.586195 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.586287 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.586444 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.586837 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.586940 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":177,"threads_started":1}
I20260929 14:46:08.587967 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.582482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39645 (local address 127.0.0.1:49250)
0929 14:46:08.583220 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.583240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.583323 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.584037 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.584040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.584057 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:08.584139 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.584145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.585434 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.585436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.586560 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.586568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.587584 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.587587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.587671 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.587693 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.587762 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.587834 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":21,"thread_start_us":172,"threads_started":1}
I20260929 14:46:08.588021 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.582705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.0.98.2:39645)
0929 14:46:08.583130 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.583142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.583409 (+ 267us) server_negotiation.cc:407] Connection header received
0929 14:46:08.583485 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.583490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.583761 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.583893 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:08.584263 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.585335 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.586674 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.587461 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.587565 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.587721 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.587799 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.587886 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":133,"threads_started":1}
I20260929 14:46:08.588429 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.582370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.0.98.1:33051)
0929 14:46:08.583312 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.583326 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.583348 (+ 22us) server_negotiation.cc:407] Connection header received
0929 14:46:08.583421 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.583426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.583607 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.583745 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:08.583939 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.585049 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.586514 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.587473 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.587563 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.587711 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.588191 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.588290 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":65,"threads_started":1}
I20260929 14:46:08.588662 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.582179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33051 (local address 127.0.0.1:48864)
0929 14:46:08.582792 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.582805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.582892 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.583735 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.583738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.583755 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:08.583856 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.583863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.585187 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.585191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.586407 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.586417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.587773 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.587777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.587863 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.587884 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.587947 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.588010 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":12,"thread_start_us":97,"threads_started":1}
I20260929 14:46:08.590704 19860 tablet_service.cc:1467] Processing CreateTablet for tablet f7ec308726aa45d296270e5304901a5d (DEFAULT_TABLE table=test-table [id=0c9c28876dc2414b8e30a8034a2394aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:08.590705 19994 tablet_service.cc:1467] Processing CreateTablet for tablet f7ec308726aa45d296270e5304901a5d (DEFAULT_TABLE table=test-table [id=0c9c28876dc2414b8e30a8034a2394aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:08.591917 19860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ec308726aa45d296270e5304901a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:08.591917 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ec308726aa45d296270e5304901a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:08.599006 19726 tablet_service.cc:1467] Processing CreateTablet for tablet f7ec308726aa45d296270e5304901a5d (DEFAULT_TABLE table=test-table [id=0c9c28876dc2414b8e30a8034a2394aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:08.599937 20077 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Bootstrap starting.
I20260929 14:46:08.600147 19726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ec308726aa45d296270e5304901a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:08.600463 20078 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Bootstrap starting.
I20260929 14:46:08.602780 20077 tablet_bootstrap.cc:654] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:08.602801 20078 tablet_bootstrap.cc:654] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:08.603977 20077 log.cc:824] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:08.604033 20078 log.cc:824] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:08.606149 20078 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: No bootstrap required, opened a new log
I20260929 14:46:08.606312 20077 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: No bootstrap required, opened a new log
I20260929 14:46:08.606343 20078 ts_tablet_manager.cc:1397] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:46:08.606511 20077 ts_tablet_manager.cc:1397] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:46:08.608568 20081 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: Bootstrap starting.
I20260929 14:46:08.609441 20078 raft_consensus.cc:348] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.609678 20078 raft_consensus.cc:374] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:08.609730 20078 raft_consensus.cc:729] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00473f8181b046db8c2031b7fe8463ca, State: Initialized, Role: FOLLOWER
I20260929 14:46:08.610141 20078 consensus_queue.cc:260] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.610168 20077 raft_consensus.cc:348] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.610428 20077 raft_consensus.cc:374] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:08.610492 20077 raft_consensus.cc:729] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904584b9309546dba1e49d9985874e67, State: Initialized, Role: FOLLOWER
I20260929 14:46:08.610957 20081 tablet_bootstrap.cc:654] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:08.610952 20077 consensus_queue.cc:260] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.611104 20078 ts_tablet_manager.cc:1428] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260929 14:46:08.611760 20060 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
I20260929 14:46:08.611900 20077 ts_tablet_manager.cc:1428] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:46:08.611886 20081 log.cc:824] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:08.612236 19926 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
I20260929 14:46:08.613988 20081 tablet_bootstrap.cc:492] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: No bootstrap required, opened a new log
I20260929 14:46:08.614195 20081 ts_tablet_manager.cc:1397] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:46:08.617895 20081 raft_consensus.cc:348] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.618186 20081 raft_consensus.cc:374] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:08.618316 20081 raft_consensus.cc:729] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f58a6203861417abc05101ca0114750, State: Initialized, Role: FOLLOWER
I20260929 14:46:08.618752 20081 consensus_queue.cc:260] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.619433 19792 heartbeater.cc:503] Master 127.0.98.62:41293 was elected leader, sending a full tablet report...
I20260929 14:46:08.619729 20081 ts_tablet_manager.cc:1428] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 14:46:08.647261 20083 raft_consensus.cc:482] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:08.647464 20083 raft_consensus.cc:504] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.648653 20083 leader_election.cc:288] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00473f8181b046db8c2031b7fe8463ca (127.0.98.3:33197), 4f58a6203861417abc05101ca0114750 (127.0.98.1:33051)
I20260929 14:46:08.652415 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.648935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33197 (local address 127.0.98.2:50213)
0929 14:46:08.649069 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.649085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.649186 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.649652 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.649654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.649697 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:08.649794 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.649799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.650576 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.650578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.651350 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.651358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.652018 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.652021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.652122 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.652139 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.652204 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.652276 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"spinlock_wait_cycles":9216}
I20260929 14:46:08.652415 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.648999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50213 (local address 127.0.98.3:33197)
0929 14:46:08.649147 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.649152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.649170 (+ 18us) server_negotiation.cc:407] Connection header received
0929 14:46:08.649305 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.649308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.649362 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.649438 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:08.649910 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.650450 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.651487 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.651897 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.651993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.652066 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.652234 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.652304 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 14:46:08.652688 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.649215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33051 (local address 127.0.98.2:58727)
0929 14:46:08.649573 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.649584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:08.649701 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:08.649941 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:08.649944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:08.649958 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:08.650072 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.650077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.650856 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.650859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.651654 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:08.651662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.652287 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.652291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:08.652385 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.652406 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 14:46:08.652470 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 14:46:08.652550 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"spinlock_wait_cycles":30720,"thread_start_us":136,"threads_started":1}
I20260929 14:46:08.652674 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.649361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58727 (local address 127.0.98.1:33051)
0929 14:46:08.649488 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.649492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.649655 (+ 163us) server_negotiation.cc:407] Connection header received
0929 14:46:08.649804 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.649807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.649851 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.649915 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:08.650167 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.650754 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.651774 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.652177 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.652269 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.652328 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.652496 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.652564 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 14:46:08.653340 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ec308726aa45d296270e5304901a5d" candidate_uuid: "904584b9309546dba1e49d9985874e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f58a6203861417abc05101ca0114750" is_pre_election: true
I20260929 14:46:08.653520 20014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ec308726aa45d296270e5304901a5d" candidate_uuid: "904584b9309546dba1e49d9985874e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00473f8181b046db8c2031b7fe8463ca" is_pre_election: true
I20260929 14:46:08.653646 19746 raft_consensus.cc:2454] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 904584b9309546dba1e49d9985874e67 in term 0.
I20260929 14:46:08.653860 20014 raft_consensus.cc:2454] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 904584b9309546dba1e49d9985874e67 in term 0.
I20260929 14:46:08.654106 19816 leader_election.cc:302] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f58a6203861417abc05101ca0114750, 904584b9309546dba1e49d9985874e67; no voters:
I20260929 14:46:08.654381 20083 raft_consensus.cc:2781] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:08.654460 20083 raft_consensus.cc:482] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:08.654510 20083 raft_consensus.cc:3037] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:08.655616 20083 raft_consensus.cc:504] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.655941 20083 leader_election.cc:288] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [CANDIDATE]: Term 1 election: Requested vote from peers 00473f8181b046db8c2031b7fe8463ca (127.0.98.3:33197), 4f58a6203861417abc05101ca0114750 (127.0.98.1:33051)
I20260929 14:46:08.656409 20014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ec308726aa45d296270e5304901a5d" candidate_uuid: "904584b9309546dba1e49d9985874e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00473f8181b046db8c2031b7fe8463ca"
I20260929 14:46:08.656450 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ec308726aa45d296270e5304901a5d" candidate_uuid: "904584b9309546dba1e49d9985874e67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f58a6203861417abc05101ca0114750"
I20260929 14:46:08.656531 20014 raft_consensus.cc:3037] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:08.656563 19746 raft_consensus.cc:3037] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:08.657680 20014 raft_consensus.cc:2454] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 904584b9309546dba1e49d9985874e67 in term 1.
I20260929 14:46:08.657727 19746 raft_consensus.cc:2454] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 904584b9309546dba1e49d9985874e67 in term 1.
I20260929 14:46:08.657964 19816 leader_election.cc:302] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00473f8181b046db8c2031b7fe8463ca, 904584b9309546dba1e49d9985874e67; no voters:
I20260929 14:46:08.658123 20083 raft_consensus.cc:2781] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:08.658372 20083 raft_consensus.cc:686] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 1 LEADER]: Becoming Leader. State: Replica: 904584b9309546dba1e49d9985874e67, State: Running, Role: LEADER
I20260929 14:46:08.658712 20083 consensus_queue.cc:237] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } }
I20260929 14:46:08.662137 19598 catalog_manager.cc:5495] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 904584b9309546dba1e49d9985874e67 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "904584b9309546dba1e49d9985874e67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "904584b9309546dba1e49d9985874e67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39645 } health_report { overall_health: HEALTHY } } }
I20260929 14:46:08.688383 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.684389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.0.98.2:39645)
0929 14:46:08.684523 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.684527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.684540 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:08.684617 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.684619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.684675 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.684765 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:08.685083 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.685606 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.686639 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.686881 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.686980 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.687270 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:08.687825 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:08.687947 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.688141 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.688230 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:46:08.694756 20014 raft_consensus.cc:1261] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 1 FOLLOWER]: Refusing update from remote peer 904584b9309546dba1e49d9985874e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:08.694756 19746 raft_consensus.cc:1261] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 1 FOLLOWER]: Refusing update from remote peer 904584b9309546dba1e49d9985874e67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:08.695470 20083 consensus_queue.cc:1035] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58a6203861417abc05101ca0114750" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:08.695675 20087 consensus_queue.cc:1035] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00473f8181b046db8c2031b7fe8463ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:08.703972 20092 mvcc.cc:205] Tried to move back new op lower bound from 7334679218966192128 to 7334679218829074432. Current Snapshot: MvccSnapshot[applied={T|T < 7334679218966192128}]
I20260929 14:46:08.713936 19601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:08.714164 19601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:08.716964 19601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1f9700d04df4363a5fa9a2bbe5bf643: Sending DeleteTablet for 3 replicas of tablet f7ec308726aa45d296270e5304901a5d
I20260929 14:46:08.717928 19726 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ec308726aa45d296270e5304901a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48864
I20260929 14:46:08.717952 19994 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ec308726aa45d296270e5304901a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60394
I20260929 14:46:08.718322 19860 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ec308726aa45d296270e5304901a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49250
I20260929 14:46:08.718495 20100 tablet_replica.cc:330] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750: stopping tablet replica
I20260929 14:46:08.718495 20099 tablet_replica.cc:330] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: stopping tablet replica
I20260929 14:46:08.718840 20100 raft_consensus.cc:2229] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:08.718840 20099 raft_consensus.cc:2229] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:08.719195 20100 raft_consensus.cc:2258] T f7ec308726aa45d296270e5304901a5d P 4f58a6203861417abc05101ca0114750 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:08.719195 20099 raft_consensus.cc:2258] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:08.719681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19663
I20260929 14:46:08.720815 20099 ts_tablet_manager.cc:1905] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:08.723196 20099 ts_tablet_manager.cc:1918] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:08.723269 20099 log.cc:1196] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/f7ec308726aa45d296270e5304901a5d
I20260929 14:46:08.723502 20099 ts_tablet_manager.cc:1939] T f7ec308726aa45d296270e5304901a5d P 00473f8181b046db8c2031b7fe8463ca: Deleting consensus metadata
I20260929 14:46:08.723824 20101 tablet_replica.cc:330] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: stopping tablet replica
I20260929 14:46:08.724155 20101 raft_consensus.cc:2229] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:46:08.724604 20101 raft_consensus.cc:2258] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:08.725024 19588 catalog_manager.cc:4841] TS 00473f8181b046db8c2031b7fe8463ca (127.0.98.3:33197): tablet f7ec308726aa45d296270e5304901a5d (table test-table [id=0c9c28876dc2414b8e30a8034a2394aa]) successfully deleted
I20260929 14:46:08.726547 20101 ts_tablet_manager.cc:1905] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:46:08.728108 19588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33051 (error 108)
I20260929 14:46:08.729184 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.728879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33051 (local address 127.0.0.1:48868)
0929 14:46:08.729015 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.729091 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 14:46:08.729183 20101 ts_tablet_manager.cc:1918] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:08.729218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19797
I20260929 14:46:08.729272 20101 log.cc:1196] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790692915364895-392-0/minicluster-data/ts-1/wal/wals/f7ec308726aa45d296270e5304901a5d
I20260929 14:46:08.729575 20101 ts_tablet_manager.cc:1939] T f7ec308726aa45d296270e5304901a5d P 904584b9309546dba1e49d9985874e67: Deleting consensus metadata
W20260929 14:46:08.730612 19588 catalog_manager.cc:4584] TS 4f58a6203861417abc05101ca0114750 (127.0.98.1:33051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7ec308726aa45d296270e5304901a5d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33051: connect: Connection refused (error 111)
I20260929 14:46:08.734408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19931
I20260929 14:46:08.735606 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.734727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39645 (local address 127.0.0.1:49258)
0929 14:46:08.735452 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:08.735513 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638}
I20260929 14:46:08.739725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 19569
2026-09-29T14:46:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2293 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T14:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:08Z Disabled control of system clock
I20260929 14:46:08.768824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42985
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:08.879521 20106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:08.879786 20106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:08.879822 20106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:08.883898 20106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:08.883963 20106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:08.883977 20106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:08.883993 20106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:08.884008 20106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:08.888885 20106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:08.889978 20106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:08.891441 20106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:08.900812 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:08.900854 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:08.900914 20106 server_base.cc:1034] running on GCE node
W20260929 14:46:08.900803 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:08.901746 20106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:08.902871 20106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:08.904062 20106 hybrid_clock.cc:648] HybridClock initialized: now 1790693168904038 us; error 34 us; skew 500 ppm
I20260929 14:46:08.906203 20106 webserver.cc:460] Webserver started at http://127.0.98.62:34439/ using document root <none> and password file <none>
I20260929 14:46:08.906765 20106 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:08.906826 20106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:08.906992 20106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:08.909174 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "7631e4603e6f400eb6bb14b1af1a8e65"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "d913eebd070abe1073cd3bf581484279"
server_key_iv: "41029bb2a1d4317e66351e60796fd813"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.909741 20106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "7631e4603e6f400eb6bb14b1af1a8e65"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "d913eebd070abe1073cd3bf581484279"
server_key_iv: "41029bb2a1d4317e66351e60796fd813"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.913607 20106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:08.916363 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:08.917784 20106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:46:08.917914 20106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "7631e4603e6f400eb6bb14b1af1a8e65"
format_stamp: "Formatted at 2026-09-29 14:46:08 on dist-test-slave-g8hp"
server_key: "d913eebd070abe1073cd3bf581484279"
server_key_iv: "41029bb2a1d4317e66351e60796fd813"
server_key_version: "encryptionkey@0"
I20260929 14:46:08.918052 20106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:08.934741 20106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:08.935478 20106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:08.935660 20106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:08.942241 20106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42985
I20260929 14:46:08.942256 20172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42985 every 8 connection(s)
I20260929 14:46:08.943209 20106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:08.945091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20106
I20260929 14:46:08.945212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:08.945475 392 external_mini_cluster.cc:1428] Setting key f339c4972d20943a59e711dfab626853
I20260929 14:46:08.946785 20173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:08.954103 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: Bootstrap starting.
I20260929 14:46:08.956631 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:08.946876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.0.98.62:42985)
0929 14:46:08.947608 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:08.947629 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:08.947674 (+ 45us) server_negotiation.cc:407] Connection header received
0929 14:46:08.948534 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:08.948557 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:08.949122 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:08.949529 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:08.950003 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.951462 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.952414 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:08.953180 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:08.953343 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:08.954433 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:08.954439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:08.954456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:08.954957 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:08.955046 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:08.955076 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:08.955168 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:08.955330 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 14:46:08.955649 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":139,"threads_started":1}
I20260929 14:46:08.956993 20173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:08.957892 20173 log.cc:824] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:08.960299 20173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: No bootstrap required, opened a new log
I20260929 14:46:08.964062 20173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } }
I20260929 14:46:08.964308 20173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:08.964341 20173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7631e4603e6f400eb6bb14b1af1a8e65, State: Initialized, Role: FOLLOWER
I20260929 14:46:08.964793 20173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } }
I20260929 14:46:08.964979 20173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:08.965052 20173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:08.965157 20173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:08.966148 20173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } }
I20260929 14:46:08.966512 20173 leader_election.cc:302] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7631e4603e6f400eb6bb14b1af1a8e65; no voters:
I20260929 14:46:08.966843 20173 leader_election.cc:288] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:08.966990 20178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:08.967247 20178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [term 1 LEADER]: Becoming Leader. State: Replica: 7631e4603e6f400eb6bb14b1af1a8e65, State: Running, Role: LEADER
I20260929 14:46:08.967559 20178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } }
I20260929 14:46:08.968118 20173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:08.969487 20180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7631e4603e6f400eb6bb14b1af1a8e65. Latest consensus state: current_term: 1 leader_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } } }
I20260929 14:46:08.969622 20180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:08.969457 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7631e4603e6f400eb6bb14b1af1a8e65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42985 } } }
I20260929 14:46:08.969831 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:08.970019 20187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:08.973369 20187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:08.979955 20187 catalog_manager.cc:1347] Generated new cluster ID: e7c3ef66cb054dcf81c5b341a0c40144
I20260929 14:46:08.980050 20187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:08.996953 20187 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:08.998256 20187 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:09.009989 20187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: Generated new TSK 0
I20260929 14:46:09.010900 20187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:09.019451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 14:46:09.133731 20197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:09.134034 20197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:09.134088 20197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:09.138362 20197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:09.138446 20197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:09.138527 20197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 14:46:09.143885 20197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:09.145361 20197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:09.146903 20197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:09.157212 20205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:09.157204 20203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:09.157328 20197 server_base.cc:1034] running on GCE node
W20260929 14:46:09.157292 20202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:09.157949 20197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:09.158651 20197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:09.159859 20197 hybrid_clock.cc:648] HybridClock initialized: now 1790693169159836 us; error 33 us; skew 500 ppm
I20260929 14:46:09.162029 20197 webserver.cc:460] Webserver started at http://127.0.98.1:36693/ using document root <none> and password file <none>
I20260929 14:46:09.162622 20197 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:09.162696 20197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:09.162895 20197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:09.165122 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data/instance:
uuid: "3bfbfd815c19458d8f891b069a7d4d28"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "efa73cfb4bd59d9f00916e6dcfc11273"
server_key_iv: "bcd281a9e8bcbfe1a9dd522c79c9cd2b"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.165681 20197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3bfbfd815c19458d8f891b069a7d4d28"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "efa73cfb4bd59d9f00916e6dcfc11273"
server_key_iv: "bcd281a9e8bcbfe1a9dd522c79c9cd2b"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.169473 20197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:09.172130 20211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.173540 20197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:09.173647 20197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal
uuid: "3bfbfd815c19458d8f891b069a7d4d28"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "efa73cfb4bd59d9f00916e6dcfc11273"
server_key_iv: "bcd281a9e8bcbfe1a9dd522c79c9cd2b"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.173750 20197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:09.190418 20197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:09.191139 20197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:09.191303 20197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:09.191908 20197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:09.192910 20197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:09.192965 20197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.193005 20197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:09.193024 20197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.202118 20197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42829
I20260929 14:46:09.202147 20324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42829 every 8 connection(s)
I20260929 14:46:09.203058 20197 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/data/info.pb
I20260929 14:46:09.206239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20197
I20260929 14:46:09.206346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-0/wal/instance
I20260929 14:46:09.206647 392 external_mini_cluster.cc:1428] Setting key c58d16d161ffb7b52abb4447e5eb3859
I20260929 14:46:09.208612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:09.213719 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.204991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36343 (local address 127.0.98.62:42985)
0929 14:46:09.205116 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.205119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.206238 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 14:46:09.207919 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.207921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.208006 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.208087 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:09.209123 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.209853 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.211535 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.211766 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.211847 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.213166 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:09.213171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:09.213174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:09.213188 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:09.213213 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:09.213222 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:09.213309 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.213489 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.213562 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:46:09.214998 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.205368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42985 (local address 127.0.98.1:36343)
0929 14:46:09.206064 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.206106 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.207718 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.208269 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.208277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.208378 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:09.208988 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.209004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.210057 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.210061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.211397 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.211409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.211921 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.211924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.212128 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.212751 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:09.212754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:09.212814 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:09.212928 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:09.213319 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:09.213330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:09.213336 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.213465 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.213912 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
I20260929 14:46:09.216349 20325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42985
I20260929 14:46:09.216601 20325 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:09.217212 20325 heartbeater.cc:511] Master 127.0.98.62:42985 requested a full tablet report, sending...
I20260929 14:46:09.218734 20137 ts_manager.cc:194] Registered new tserver with Master: 3bfbfd815c19458d8f891b069a7d4d28 (127.0.98.1:42829)
I20260929 14:46:09.219866 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36343
W20260929 14:46:09.322333 20328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:09.322635 20328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:09.322679 20328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:09.326972 20328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:09.327045 20328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:09.327103 20328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 14:46:09.331993 20328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:09.333086 20328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:09.334474 20328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:09.344681 20334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:09.344698 20333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:09.344760 20328 server_base.cc:1034] running on GCE node
W20260929 14:46:09.344698 20336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:09.345459 20328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:09.346170 20328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:09.347352 20328 hybrid_clock.cc:648] HybridClock initialized: now 1790693169347340 us; error 32 us; skew 500 ppm
I20260929 14:46:09.349457 20328 webserver.cc:460] Webserver started at http://127.0.98.2:38769/ using document root <none> and password file <none>
I20260929 14:46:09.350062 20328 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:09.350132 20328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:09.350330 20328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:09.352494 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c10e1b2fa374313965702f3b11a1b10"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "9fa43903b31516575a293a804c684e41"
server_key_iv: "601bbd81a9ef3df7d4f6c7d2362f96d2"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.353106 20328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c10e1b2fa374313965702f3b11a1b10"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "9fa43903b31516575a293a804c684e41"
server_key_iv: "601bbd81a9ef3df7d4f6c7d2362f96d2"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.357043 20328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:09.359688 20342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.361161 20328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:09.361279 20328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal
uuid: "2c10e1b2fa374313965702f3b11a1b10"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "9fa43903b31516575a293a804c684e41"
server_key_iv: "601bbd81a9ef3df7d4f6c7d2362f96d2"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.361392 20328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:09.372917 20328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:09.373610 20328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:09.373788 20328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:09.374447 20328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:09.375454 20328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:09.375511 20328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.375566 20328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:09.375594 20328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.383946 20328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42481
I20260929 14:46:09.383968 20455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42481 every 8 connection(s)
I20260929 14:46:09.384832 20328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/data/info.pb
I20260929 14:46:09.385639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20328
I20260929 14:46:09.385763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-1/wal/instance
I20260929 14:46:09.386114 392 external_mini_cluster.cc:1428] Setting key b58e1329993f3c7d700310aa6642646b
I20260929 14:46:09.388160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--builtin_ntp_servers=127.0.98.20:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 14:46:09.395965 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.386724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36467 (local address 127.0.98.62:42985)
0929 14:46:09.386853 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.386858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.388230 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 14:46:09.390210 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.390214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.390275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.390371 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:09.391288 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.391905 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.393710 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.393933 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.394012 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.395428 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:09.395431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:09.395433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:09.395446 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:09.395467 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:09.395475 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:09.395534 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.395767 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.395837 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:46:09.397240 20458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.387226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42985 (local address 127.0.98.2:36467)
0929 14:46:09.388054 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.388113 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.389920 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.390519 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.390527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.390627 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:09.391137 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.391150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.392068 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.392072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.393560 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.393573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.394106 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.394111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.394331 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.395079 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:09.395082 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:09.395146 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:09.395247 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:09.395561 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:09.395571 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:09.395578 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.395720 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.396078 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":116,"threads_started":1}
I20260929 14:46:09.398613 20456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42985
I20260929 14:46:09.398866 20456 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:09.399448 20456 heartbeater.cc:511] Master 127.0.98.62:42985 requested a full tablet report, sending...
I20260929 14:46:09.400534 20137 ts_manager.cc:194] Registered new tserver with Master: 2c10e1b2fa374313965702f3b11a1b10 (127.0.98.2:42481)
I20260929 14:46:09.401229 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36467
W20260929 14:46:09.508298 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:09.508611 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:09.508668 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:09.513077 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:09.513147 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:09.513237 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 14:46:09.518271 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:09.519393 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:09.520823 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:09.532694 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:09.532756 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:09.532927 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:09.533106 20459 server_base.cc:1034] running on GCE node
I20260929 14:46:09.533622 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:09.534315 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:09.535516 20459 hybrid_clock.cc:648] HybridClock initialized: now 1790693169535474 us; error 43 us; skew 500 ppm
I20260929 14:46:09.537631 20459 webserver.cc:460] Webserver started at http://127.0.98.3:46557/ using document root <none> and password file <none>
I20260929 14:46:09.538199 20459 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:09.538269 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:09.538476 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:09.540669 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e89f74e4fa44f6689a9d703e7334807"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "dfb339e43c93bbd880b28d402e87d472"
server_key_iv: "931e6f6974c1529a78546a9499020a93"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.541303 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e89f74e4fa44f6689a9d703e7334807"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "dfb339e43c93bbd880b28d402e87d472"
server_key_iv: "931e6f6974c1529a78546a9499020a93"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.545255 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:09.548081 20473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.549750 20459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:09.549904 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal
uuid: "0e89f74e4fa44f6689a9d703e7334807"
format_stamp: "Formatted at 2026-09-29 14:46:09 on dist-test-slave-g8hp"
server_key: "dfb339e43c93bbd880b28d402e87d472"
server_key_iv: "931e6f6974c1529a78546a9499020a93"
server_key_version: "encryptionkey@0"
I20260929 14:46:09.550025 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:09.564500 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:09.565383 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:09.565594 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:09.566249 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:46:09.567317 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:46:09.567368 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.567423 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:46:09.567451 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:09.576525 20459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38561
I20260929 14:46:09.576625 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38561 every 8 connection(s)
I20260929 14:46:09.577603 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/data/info.pb
I20260929 14:46:09.586097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20459
I20260929 14:46:09.586205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal/instance
I20260929 14:46:09.586483 392 external_mini_cluster.cc:1428] Setting key f59913ce16b991f2aa98a76a04adfe58
I20260929 14:46:09.587658 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.579599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45569 (local address 127.0.98.62:42985)
0929 14:46:09.579770 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.579775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.580847 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 14:46:09.582290 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.582293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.582345 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.582419 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:09.583416 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.583989 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.585373 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.585610 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.585694 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.586933 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:09.586937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:09.586940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:09.586956 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:09.586991 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:09.587002 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:09.587074 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.587395 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.587478 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 14:46:09.588941 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.579936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42985 (local address 127.0.98.3:45569)
0929 14:46:09.580687 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.580731 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.582069 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.582649 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.582656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.582748 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:09.583267 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.583278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.584124 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.584127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.585238 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.585250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.585776 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.585780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.585944 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.586645 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 14:46:09.586647 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:09.586692 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 14:46:09.586767 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:09.587190 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:09.587203 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:09.587209 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.587364 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.587722 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260929 14:46:09.590198 20587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42985
I20260929 14:46:09.590441 20587 heartbeater.cc:464] Registering TS with master...
I20260929 14:46:09.590993 20587 heartbeater.cc:511] Master 127.0.98.62:42985 requested a full tablet report, sending...
I20260929 14:46:09.592087 20137 ts_manager.cc:194] Registered new tserver with Master: 0e89f74e4fa44f6689a9d703e7334807 (127.0.98.3:38561)
I20260929 14:46:09.592630 20137 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45569
I20260929 14:46:09.601063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 14:46:09.607026 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.603086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.0.98.62:42985)
0929 14:46:09.603200 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.603203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.603376 (+ 173us) server_negotiation.cc:407] Connection header received
0929 14:46:09.603556 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.603558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.603602 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.603672 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:09.603941 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.604584 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.605512 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.605729 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.605821 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.606651 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:09.606654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:09.606656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:09.606669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:09.606689 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:09.606697 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:09.606763 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.606851 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.606907 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 14:46:09.610075 20137 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:46:09.613133 20137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:46:09.628952 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.623192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.98.3:38561)
0929 14:46:09.623630 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.623641 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.624112 (+ 471us) server_negotiation.cc:407] Connection header received
0929 14:46:09.624354 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.624361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.624593 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.624738 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:09.625282 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.626069 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.627645 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.628218 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.628326 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.628562 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.628650 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.628750 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":176,"threads_started":1}
I20260929 14:46:09.628953 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.622574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38561 (local address 127.0.0.1:46448)
0929 14:46:09.624005 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.624019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.624185 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.624966 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.624968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.625002 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:09.625168 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.625174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.626190 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.626193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.627543 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.627554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.628341 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.628344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.628458 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.628497 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.628610 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.628741 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"mutex_wait_us":97,"spinlock_wait_cycles":3968,"thread_start_us":78,"threads_started":1}
I20260929 14:46:09.629282 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.622486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42481 (local address 127.0.0.1:51460)
0929 14:46:09.623955 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.624013 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.624152 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.625097 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.625099 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.625113 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:09.625190 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.625196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.626208 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.626211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.627483 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.627493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.628837 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.628841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.628964 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.628988 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.629064 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.629144 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1283,"mutex_wait_us":2,"thread_start_us":166,"threads_started":1}
I20260929 14:46:09.629733 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.622637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.2:42481)
0929 14:46:09.623001 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.623014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.624114 (+ 1100us) server_negotiation.cc:407] Connection header received
0929 14:46:09.624625 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.624629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.624782 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.624952 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:09.625304 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.626092 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.627625 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.628485 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.628595 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.628777 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.629466 (+ 689us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.629572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":144,"threads_started":1}
I20260929 14:46:09.630893 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.623362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.0.1:42354)
0929 14:46:09.624375 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.624387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.624479 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.624955 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.624962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.625000 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:09.625158 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.625165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.627552 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.627556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.628702 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.628712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.630546 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.630549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.630638 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.630661 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.630718 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.630789 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"spinlock_wait_cycles":6272,"thread_start_us":99,"threads_started":1}
I20260929 14:46:09.631430 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.623616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.0.98.1:42829)
0929 14:46:09.623907 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.623932 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.624476 (+ 544us) server_negotiation.cc:407] Connection header received
0929 14:46:09.624580 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.624585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.624763 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.624958 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:09.626209 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.627435 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.628837 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.630008 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.630279 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.631168 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.631211 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.631281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260929 14:46:09.632225 20521 tablet_service.cc:1467] Processing CreateTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 (DEFAULT_TABLE table=test-table [id=12a62cffc31e458cb86b00e5848d1314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:09.633280 20521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dff527e0da146b2b3822fc2ec9a0108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:09.633642 20390 tablet_service.cc:1467] Processing CreateTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 (DEFAULT_TABLE table=test-table [id=12a62cffc31e458cb86b00e5848d1314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:09.634229 20259 tablet_service.cc:1467] Processing CreateTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 (DEFAULT_TABLE table=test-table [id=12a62cffc31e458cb86b00e5848d1314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 14:46:09.634748 20390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dff527e0da146b2b3822fc2ec9a0108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:09.635265 20259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dff527e0da146b2b3822fc2ec9a0108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:09.639793 20604 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Bootstrap starting.
I20260929 14:46:09.642185 20604 tablet_bootstrap.cc:654] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:09.643175 20604 log.cc:824] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:09.643435 20606 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Bootstrap starting.
I20260929 14:46:09.643873 20605 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Bootstrap starting.
I20260929 14:46:09.645502 20604 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: No bootstrap required, opened a new log
I20260929 14:46:09.645715 20605 tablet_bootstrap.cc:654] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:09.645730 20604 ts_tablet_manager.cc:1397] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:46:09.645766 20606 tablet_bootstrap.cc:654] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:09.646508 20605 log.cc:824] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:09.646605 20606 log.cc:824] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:09.648926 20605 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: No bootstrap required, opened a new log
I20260929 14:46:09.648938 20606 tablet_bootstrap.cc:492] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: No bootstrap required, opened a new log
I20260929 14:46:09.649152 20605 ts_tablet_manager.cc:1397] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:46:09.649250 20606 ts_tablet_manager.cc:1397] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:46:09.650298 20604 raft_consensus.cc:348] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.650596 20604 raft_consensus.cc:374] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:09.650660 20604 raft_consensus.cc:729] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e89f74e4fa44f6689a9d703e7334807, State: Initialized, Role: FOLLOWER
I20260929 14:46:09.651185 20604 consensus_queue.cc:260] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.651923 20587 heartbeater.cc:503] Master 127.0.98.62:42985 was elected leader, sending a full tablet report...
I20260929 14:46:09.652505 20604 ts_tablet_manager.cc:1428] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:46:09.653470 20605 raft_consensus.cc:348] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.653690 20605 raft_consensus.cc:374] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:09.653740 20605 raft_consensus.cc:729] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bfbfd815c19458d8f891b069a7d4d28, State: Initialized, Role: FOLLOWER
I20260929 14:46:09.653748 20606 raft_consensus.cc:348] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.654035 20606 raft_consensus.cc:374] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:09.654093 20606 raft_consensus.cc:729] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c10e1b2fa374313965702f3b11a1b10, State: Initialized, Role: FOLLOWER
I20260929 14:46:09.654273 20605 consensus_queue.cc:260] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.654511 20606 consensus_queue.cc:260] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.655166 20456 heartbeater.cc:503] Master 127.0.98.62:42985 was elected leader, sending a full tablet report...
I20260929 14:46:09.655431 20606 ts_tablet_manager.cc:1428] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:46:09.655431 20605 ts_tablet_manager.cc:1428] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:46:09.655782 20325 heartbeater.cc:503] Master 127.0.98.62:42985 was elected leader, sending a full tablet report...
W20260929 14:46:09.704412 20326 tablet.cc:2369] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:09.769315 20610 raft_consensus.cc:482] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:46:09.769616 20610 raft_consensus.cc:504] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.770800 20610 leader_election.cc:288] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c10e1b2fa374313965702f3b11a1b10 (127.0.98.2:42481), 3bfbfd815c19458d8f891b069a7d4d28 (127.0.98.1:42829)
I20260929 14:46:09.774935 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.771098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42481 (local address 127.0.98.3:39257)
0929 14:46:09.771227 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.771243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.771355 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.771661 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.771664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.771685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:09.771833 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.771840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.772746 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.772750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.773654 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.773664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.774427 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.774431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.774601 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.774630 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.774697 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.774784 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3,"spinlock_wait_cycles":2304}
I20260929 14:46:09.774981 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.771191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39257 (local address 127.0.98.2:42481)
0929 14:46:09.771311 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.771316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.771348 (+ 32us) server_negotiation.cc:407] Connection header received
0929 14:46:09.771457 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.771485 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.771544 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.771640 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:09.772006 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.772629 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.773784 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.774305 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.774388 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.774446 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.774722 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.774805 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:46:09.775077 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.771272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34851 (local address 127.0.98.1:42829)
0929 14:46:09.771396 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.771399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.771714 (+ 315us) server_negotiation.cc:407] Connection header received
0929 14:46:09.771837 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.771840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.771896 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.771970 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 14:46:09.772279 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.772821 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.774099 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.774551 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.774660 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.774718 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.774890 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.774962 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:46:09.775564 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.771098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.98.3:34851)
0929 14:46:09.771611 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.771625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:09.771737 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:09.772016 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:09.772019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:09.772038 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 14:46:09.772164 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.772170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.773002 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.773005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.773968 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:09.773979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.774671 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.774675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:09.774772 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.774794 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 14:46:09.774866 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 14:46:09.774928 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":135,"spinlock_wait_cycles":22144,"thread_start_us":161,"threads_started":1}
I20260929 14:46:09.775651 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dff527e0da146b2b3822fc2ec9a0108" candidate_uuid: "0e89f74e4fa44f6689a9d703e7334807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c10e1b2fa374313965702f3b11a1b10" is_pre_election: true
I20260929 14:46:09.775995 20410 raft_consensus.cc:2454] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e89f74e4fa44f6689a9d703e7334807 in term 0.
I20260929 14:46:09.776160 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dff527e0da146b2b3822fc2ec9a0108" candidate_uuid: "0e89f74e4fa44f6689a9d703e7334807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfbfd815c19458d8f891b069a7d4d28" is_pre_election: true
I20260929 14:46:09.776513 20279 raft_consensus.cc:2454] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e89f74e4fa44f6689a9d703e7334807 in term 0.
I20260929 14:46:09.776546 20476 leader_election.cc:302] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e89f74e4fa44f6689a9d703e7334807, 2c10e1b2fa374313965702f3b11a1b10; no voters:
I20260929 14:46:09.776844 20610 raft_consensus.cc:2781] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:46:09.776963 20610 raft_consensus.cc:482] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:46:09.777016 20610 raft_consensus.cc:3037] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:09.778169 20610 raft_consensus.cc:504] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.778501 20610 leader_election.cc:288] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [CANDIDATE]: Term 1 election: Requested vote from peers 2c10e1b2fa374313965702f3b11a1b10 (127.0.98.2:42481), 3bfbfd815c19458d8f891b069a7d4d28 (127.0.98.1:42829)
I20260929 14:46:09.778891 20410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dff527e0da146b2b3822fc2ec9a0108" candidate_uuid: "0e89f74e4fa44f6689a9d703e7334807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c10e1b2fa374313965702f3b11a1b10"
I20260929 14:46:09.779018 20410 raft_consensus.cc:3037] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:09.780210 20410 raft_consensus.cc:2454] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e89f74e4fa44f6689a9d703e7334807 in term 1.
I20260929 14:46:09.780501 20476 leader_election.cc:302] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e89f74e4fa44f6689a9d703e7334807, 2c10e1b2fa374313965702f3b11a1b10; no voters:
I20260929 14:46:09.780491 20279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dff527e0da146b2b3822fc2ec9a0108" candidate_uuid: "0e89f74e4fa44f6689a9d703e7334807" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bfbfd815c19458d8f891b069a7d4d28"
I20260929 14:46:09.780619 20279 raft_consensus.cc:3037] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:09.780668 20610 raft_consensus.cc:2781] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:09.780903 20610 raft_consensus.cc:686] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 1 LEADER]: Becoming Leader. State: Replica: 0e89f74e4fa44f6689a9d703e7334807, State: Running, Role: LEADER
I20260929 14:46:09.781234 20610 consensus_queue.cc:237] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } }
I20260929 14:46:09.781905 20279 raft_consensus.cc:2454] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e89f74e4fa44f6689a9d703e7334807 in term 1.
I20260929 14:46:09.784478 20136 catalog_manager.cc:5495] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e89f74e4fa44f6689a9d703e7334807 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e89f74e4fa44f6689a9d703e7334807" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e89f74e4fa44f6689a9d703e7334807" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 } health_report { overall_health: UNKNOWN } } }
I20260929 14:46:09.825887 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.821595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.0.98.3:38561)
0929 14:46:09.821725 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:09.821728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:09.821741 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:09.821828 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:09.821830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:09.821884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:09.821986 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:09.822257 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.822852 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.823923 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:09.824226 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:09.824360 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:09.824785 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:09.825323 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:09.825439 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:09.825657 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 14:46:09.825716 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 14:46:09.828517 20588 tablet.cc:2369] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:09.832255 20279 raft_consensus.cc:1261] T 2dff527e0da146b2b3822fc2ec9a0108 P 3bfbfd815c19458d8f891b069a7d4d28 [term 1 FOLLOWER]: Refusing update from remote peer 0e89f74e4fa44f6689a9d703e7334807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:09.832255 20410 raft_consensus.cc:1261] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 1 FOLLOWER]: Refusing update from remote peer 0e89f74e4fa44f6689a9d703e7334807: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:09.833021 20610 consensus_queue.cc:1035] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bfbfd815c19458d8f891b069a7d4d28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:09.833247 20614 consensus_queue.cc:1035] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c10e1b2fa374313965702f3b11a1b10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:09.841801 20619 mvcc.cc:205] Tried to move back new op lower bound from 7334679223625064448 to 7334679223426322432. Current Snapshot: MvccSnapshot[applied={T|T < 7334679223625064448}]
I20260929 14:46:09.851385 20132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:09.851603 20132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 14:46:09.854363 20132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7631e4603e6f400eb6bb14b1af1a8e65: Sending DeleteTablet for 3 replicas of tablet 2dff527e0da146b2b3822fc2ec9a0108
I20260929 14:46:09.855289 20521 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:09 UTC) from {username='slave'} at 127.0.0.1:46448
I20260929 14:46:09.855302 20259 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:09 UTC) from {username='slave'} at 127.0.0.1:42354
I20260929 14:46:09.855849 20627 tablet_replica.cc:330] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: stopping tablet replica
I20260929 14:46:09.855911 20390 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dff527e0da146b2b3822fc2ec9a0108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 14:46:09 UTC) from {username='slave'} at 127.0.0.1:51460
I20260929 14:46:09.856201 20627 raft_consensus.cc:2229] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:46:09.856274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20197
I20260929 14:46:09.856606 20627 raft_consensus.cc:2258] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:09.857826 20627 ts_tablet_manager.cc:1905] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 14:46:09.860296 20627 ts_tablet_manager.cc:1918] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:09.860392 20627 log.cc:1196] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Deleting WAL directory at /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790692915364895-392-0/minicluster-data/ts-2/wal/wals/2dff527e0da146b2b3822fc2ec9a0108
I20260929 14:46:09.860651 20627 ts_tablet_manager.cc:1939] T 2dff527e0da146b2b3822fc2ec9a0108 P 0e89f74e4fa44f6689a9d703e7334807: Deleting consensus metadata
I20260929 14:46:09.861085 20629 tablet_replica.cc:330] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: stopping tablet replica
I20260929 14:46:09.861488 20629 raft_consensus.cc:2229] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:46:09.861912 20629 raft_consensus.cc:2258] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:46:09.861994 20124 catalog_manager.cc:4841] TS 0e89f74e4fa44f6689a9d703e7334807 (127.0.98.3:38561): tablet 2dff527e0da146b2b3822fc2ec9a0108 (table test-table [id=12a62cffc31e458cb86b00e5848d1314]) successfully deleted
I20260929 14:46:09.863517 20629 ts_tablet_manager.cc:1905] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 14:46:09.864773 20121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42829 (error 108)
I20260929 14:46:09.865825 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.865534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42829 (local address 127.0.0.1:42370)
0929 14:46:09.865655 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.865720 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 14:46:09.865887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20328
W20260929 14:46:09.866142 20121 catalog_manager.cc:4584] TS 3bfbfd815c19458d8f891b069a7d4d28 (127.0.98.1:42829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2dff527e0da146b2b3822fc2ec9a0108: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42829: connect: Connection refused (error 111)
I20260929 14:46:09.866248 20629 ts_tablet_manager.cc:1918] T 2dff527e0da146b2b3822fc2ec9a0108 P 2c10e1b2fa374313965702f3b11a1b10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 14:46:09.871915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20459
I20260929 14:46:09.873188 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:09.872075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42481 (local address 127.0.0.1:51472)
0929 14:46:09.873037 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:09.873098 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":864}
I20260929 14:46:09.877279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20106
2026-09-29T14:46:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1136 ms)
[----------] 6 tests from EncryptionPolicyTest (10516 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:09 dist-test-slave-g8hp krb5kdc[20633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:09 dist-test-slave-g8hp krb5kdc[20633](info): set up 2 sockets
Sep 29 14:46:09 dist-test-slave-g8hp krb5kdc[20633](info): commencing operation
krb5kdc: starting...
W20260929 14:46:10.848969 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:10 dist-test-slave-g8hp krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:10.973094 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:11.087209 20646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:11.087510 20646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:11.087554 20646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:11.091751 20646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:11.091904 20646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:11.091924 20646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:11.091941 20646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:11.091956 20646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:11.097669 20646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:11.098738 20646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:11.099752 20646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:11.108726 20651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:11.108783 20652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:11.108843 20646 server_base.cc:1034] running on GCE node
W20260929 14:46:11.108742 20654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:11.109724 20646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:11.111011 20646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:11.112205 20646 hybrid_clock.cc:648] HybridClock initialized: now 1790693171112179 us; error 40 us; skew 500 ppm
Sep 29 14:46:11 dist-test-slave-g8hp krb5kdc[20633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:11.121680 20646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:11.122815 20646 webserver.cc:460] Webserver started at http://127.0.0.1:33425/ using document root <none> and password file <none>
I20260929 14:46:11.123445 20646 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:11.123518 20646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:11.123718 20646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:11.125936 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "2d1820462cda4207a9e17350eb2f2e67"
format_stamp: "Formatted at 2026-09-29 14:46:11 on dist-test-slave-g8hp"
server_key: "c50e3e86960b1afd0e3cd5e5f8c7ca56"
server_key_iv: "65888d7b77ddb437e297ccc8ac223b5d"
server_key_version: "encryptionkey@0"
I20260929 14:46:11.126510 20646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d1820462cda4207a9e17350eb2f2e67"
format_stamp: "Formatted at 2026-09-29 14:46:11 on dist-test-slave-g8hp"
server_key: "c50e3e86960b1afd0e3cd5e5f8c7ca56"
server_key_iv: "65888d7b77ddb437e297ccc8ac223b5d"
server_key_version: "encryptionkey@0"
I20260929 14:46:11.130692 20646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:11.133363 20661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:11.134785 20646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:11.134917 20646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "2d1820462cda4207a9e17350eb2f2e67"
format_stamp: "Formatted at 2026-09-29 14:46:11 on dist-test-slave-g8hp"
server_key: "c50e3e86960b1afd0e3cd5e5f8c7ca56"
server_key_iv: "65888d7b77ddb437e297ccc8ac223b5d"
server_key_version: "encryptionkey@0"
I20260929 14:46:11.135036 20646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:11.149290 20646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:11.152733 20646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:11.152966 20646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:11.159845 20646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33613
I20260929 14:46:11.159868 20713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33613 every 8 connection(s)
I20260929 14:46:11.160832 20646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:11.163699 20714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:11.169449 20714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: Bootstrap starting.
I20260929 14:46:11.170097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20646
I20260929 14:46:11.170239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:11.170511 392 external_mini_cluster.cc:1428] Setting key ef2414acbc2130d72416ffcfd2ede07c
I20260929 14:46:11.172432 20714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:11.173219 20714 log.cc:824] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:11.175397 20714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: No bootstrap required, opened a new log
Sep 29 14:46:11 dist-test-slave-g8hp krb5kdc[20633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:46:11.179761 20714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 14:46:11.180049 20714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:11.180109 20714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1820462cda4207a9e17350eb2f2e67, State: Initialized, Role: FOLLOWER
I20260929 14:46:11.180570 20714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 14:46:11.180730 20714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:11.180802 20714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:11.180926 20714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:11.182109 20714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 14:46:11.182499 20714 leader_election.cc:302] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d1820462cda4207a9e17350eb2f2e67; no voters:
I20260929 14:46:11.182861 20714 leader_election.cc:288] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:11.183003 20719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:11.183249 20719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [term 1 LEADER]: Becoming Leader. State: Replica: 2d1820462cda4207a9e17350eb2f2e67, State: Running, Role: LEADER
I20260929 14:46:11.183621 20719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 14:46:11.184263 20714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:11.185290 20721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d1820462cda4207a9e17350eb2f2e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } } }
I20260929 14:46:11.185529 20720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d1820462cda4207a9e17350eb2f2e67. Latest consensus state: current_term: 1 leader_uuid: "2d1820462cda4207a9e17350eb2f2e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d1820462cda4207a9e17350eb2f2e67" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } } }
I20260929 14:46:11.185611 20720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:11.185609 20721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:11.187064 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:11.171903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47981 (local address 127.0.0.1:33613)
0929 14:46:11.172353 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:11.172369 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:11.172410 (+ 41us) server_negotiation.cc:407] Connection header received
0929 14:46:11.173151 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:11.173170 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:11.173558 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:11.173971 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:11.174771 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:11.175847 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:11.177041 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:11.177528 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:11.177644 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:11.180674 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:11.180680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:11.180699 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:11.180734 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:11.182684 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:11.183575 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:11.183581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:11.183587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:11.183730 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:11.184128 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:11.184131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:11.184133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:11.184677 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:11.184910 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:11.185816 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 14:46:11.186059 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":138,"threads_started":1}
W20260929 14:46:11.189312 20734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:11.189393 20734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:11.189450 20729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:11.190095 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:11.190275 20729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:11.196831 20729 catalog_manager.cc:1347] Generated new cluster ID: bd3208a8cc1547b9a2eecc3e71db5c41
I20260929 14:46:11.196969 20729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:11.200948 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:11.191981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45351 (local address 127.0.0.1:33613)
0929 14:46:11.192079 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:11.192084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:11.192485 (+ 401us) server_negotiation.cc:407] Connection header received
0929 14:46:11.192611 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:11.192614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:11.192666 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:11.192748 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:11.193590 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:11.194419 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:11.195414 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:11.195788 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:11.195899 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:11.197676 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:11.197680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:11.197683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:11.197711 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:11.199179 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:11.199671 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:11.199674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:11.199675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:11.199755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:11.200058 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:11.200061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:11.200063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:11.200240 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:11.200385 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:11.200656 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 14:46:11.200745 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 14:46:11.214670 20729 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:11.215687 20729 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:11.227298 20729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d1820462cda4207a9e17350eb2f2e67: Generated new TSK 0
I20260929 14:46:11.228092 20729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:11.236028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20646
W20260929 14:46:11.240756 20631 connection.cc:535] client connection to 127.0.0.1:33613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33613): Connection reset by peer (error 104)
2026-09-29T14:46:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:11 dist-test-slave-g8hp krb5kdc[20745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:11 dist-test-slave-g8hp krb5kdc[20745](info): set up 2 sockets
Sep 29 14:46:11 dist-test-slave-g8hp krb5kdc[20745](info): commencing operation
krb5kdc: starting...
W20260929 14:46:12.196836 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:12 dist-test-slave-g8hp krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:12.324916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:12.436542 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:12.436897 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:12.436965 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:12.441179 20758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:12.441257 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:12.441282 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:12.441305 20758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:12.441320 20758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:12.447008 20758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:12.448170 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:12.449234 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:12.458236 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:12.458230 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:12.458482 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:12.458454 20758 server_base.cc:1034] running on GCE node
I20260929 14:46:12.459220 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:12.460422 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:12.461602 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790693172461555 us; error 61 us; skew 500 ppm
Sep 29 14:46:12 dist-test-slave-g8hp krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:12.464504 20758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:12.465520 20758 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
I20260929 14:46:12.466069 20758 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:12.466135 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:12.466323 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:12.468449 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "0344d9b9c5e04fe0abc0991fb1a1319e"
format_stamp: "Formatted at 2026-09-29 14:46:12 on dist-test-slave-g8hp"
server_key: "9fbf48ce870d65060f2794b12f4478ff"
server_key_iv: "7c5fcd6e0a39ff7f3d8c0791de478511"
server_key_version: "encryptionkey@0"
I20260929 14:46:12.469113 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "0344d9b9c5e04fe0abc0991fb1a1319e"
format_stamp: "Formatted at 2026-09-29 14:46:12 on dist-test-slave-g8hp"
server_key: "9fbf48ce870d65060f2794b12f4478ff"
server_key_iv: "7c5fcd6e0a39ff7f3d8c0791de478511"
server_key_version: "encryptionkey@0"
I20260929 14:46:12.473153 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:12.475755 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:12.477252 20758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:12.477385 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "0344d9b9c5e04fe0abc0991fb1a1319e"
format_stamp: "Formatted at 2026-09-29 14:46:12 on dist-test-slave-g8hp"
server_key: "9fbf48ce870d65060f2794b12f4478ff"
server_key_iv: "7c5fcd6e0a39ff7f3d8c0791de478511"
server_key_version: "encryptionkey@0"
I20260929 14:46:12.477512 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:12.495255 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:12.498646 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:12.498842 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:12.506031 20758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39641
I20260929 14:46:12.506049 20825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39641 every 8 connection(s)
I20260929 14:46:12.507028 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:12.509980 20826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:12.511540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20758
I20260929 14:46:12.511674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:12.511955 392 external_mini_cluster.cc:1428] Setting key b59562e4ad274f2c250dbe9b056e52d5
I20260929 14:46:12.516328 20826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e: Bootstrap starting.
I20260929 14:46:12.518640 20826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:12.519375 20826 log.cc:824] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:12.521045 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:12.513386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55969 (local address 127.0.0.1:39641)
0929 14:46:12.513875 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:12.513891 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:12.513930 (+ 39us) server_negotiation.cc:407] Connection header received
0929 14:46:12.514703 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:12.514723 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:12.515095 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:12.515439 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:12.515857 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:12.516900 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:12.517782 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:12.518262 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:12.518418 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:12.519299 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:12.519304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:12.519318 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:12.519733 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:12.519793 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:12.519817 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:12.519942 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:12.520128 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 14:46:12.520267 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":121,"threads_started":1}
I20260929 14:46:12.521427 20826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e: No bootstrap required, opened a new log
I20260929 14:46:12.524230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:12.525475 20826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260929 14:46:12.525717 20826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:12.525782 20826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0344d9b9c5e04fe0abc0991fb1a1319e, State: Initialized, Role: FOLLOWER
I20260929 14:46:12.526234 20826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260929 14:46:12.526396 20826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:12.526463 20826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:12.526561 20826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:12.527501 20826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260929 14:46:12.527835 20826 leader_election.cc:302] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0344d9b9c5e04fe0abc0991fb1a1319e; no voters:
I20260929 14:46:12.528142 20826 leader_election.cc:288] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:12.528227 20836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:12.528391 20836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [term 1 LEADER]: Becoming Leader. State: Replica: 0344d9b9c5e04fe0abc0991fb1a1319e, State: Running, Role: LEADER
I20260929 14:46:12.528697 20836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } }
I20260929 14:46:12.529549 20826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:12.530345 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0344d9b9c5e04fe0abc0991fb1a1319e. Latest consensus state: current_term: 1 leader_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260929 14:46:12.530486 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:12.530682 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:12.526203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46891 (local address 127.0.0.1:39641)
0929 14:46:12.526325 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:12.526329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:12.526550 (+ 221us) server_negotiation.cc:407] Connection header received
0929 14:46:12.526683 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:12.526686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:12.526728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:12.526790 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:12.527057 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:12.527576 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:12.528484 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:12.528793 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:12.528902 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:12.529956 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:12.529960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:12.529962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:12.529977 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:12.530007 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:12.530021 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:12.530079 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:12.530401 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 14:46:12.530442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 14:46:12.530786 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0344d9b9c5e04fe0abc0991fb1a1319e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39641 } } }
I20260929 14:46:12.530872 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:12.532042 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:12.535118 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:12.540787 20845 catalog_manager.cc:1347] Generated new cluster ID: 6a52a44b87554f07bc5ecb157d0d9bd0
I20260929 14:46:12.540933 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:12.548664 20845 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:12.549975 20845 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:12.565039 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0344d9b9c5e04fe0abc0991fb1a1319e: Generated new TSK 0
I20260929 14:46:12.565806 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:12.574235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20758
W20260929 14:46:12.579101 20743 connection.cc:535] client connection to 127.0.0.1:39641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39641): Connection reset by peer (error 104)
2026-09-29T14:46:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:12 dist-test-slave-g8hp krb5kdc[20857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:12 dist-test-slave-g8hp krb5kdc[20857](info): set up 2 sockets
Sep 29 14:46:12 dist-test-slave-g8hp krb5kdc[20857](info): commencing operation
krb5kdc: starting...
W20260929 14:46:13.540890 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:13 dist-test-slave-g8hp krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:13.668944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:13.784435 20870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:13.784806 20870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:13.784852 20870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:13.789197 20870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:13.789284 20870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:13.789304 20870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:13.789320 20870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:13.789333 20870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:13.794991 20870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:13.796111 20870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:13.797210 20870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:13.806028 20876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:13.806074 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:13.806070 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:13.806123 20870 server_base.cc:1034] running on GCE node
I20260929 14:46:13.806897 20870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:13.808068 20870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:13.809271 20870 hybrid_clock.cc:648] HybridClock initialized: now 1790693173809251 us; error 43 us; skew 500 ppm
Sep 29 14:46:13 dist-test-slave-g8hp krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:13.812218 20870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:13.813218 20870 webserver.cc:460] Webserver started at http://127.0.0.1:38329/ using document root <none> and password file <none>
I20260929 14:46:13.813782 20870 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:13.813862 20870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:13.814081 20870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:13.816450 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "b5ae264007ab4a18911ac9c31b4be485"
format_stamp: "Formatted at 2026-09-29 14:46:13 on dist-test-slave-g8hp"
server_key: "cdf1fe8d88de79df0b938a3567935591"
server_key_iv: "94834a7e0bd889ed6d08286c11e2398c"
server_key_version: "encryptionkey@0"
I20260929 14:46:13.817106 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5ae264007ab4a18911ac9c31b4be485"
format_stamp: "Formatted at 2026-09-29 14:46:13 on dist-test-slave-g8hp"
server_key: "cdf1fe8d88de79df0b938a3567935591"
server_key_iv: "94834a7e0bd889ed6d08286c11e2398c"
server_key_version: "encryptionkey@0"
I20260929 14:46:13.821080 20870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:13.823715 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:13.825184 20870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:13.825313 20870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "b5ae264007ab4a18911ac9c31b4be485"
format_stamp: "Formatted at 2026-09-29 14:46:13 on dist-test-slave-g8hp"
server_key: "cdf1fe8d88de79df0b938a3567935591"
server_key_iv: "94834a7e0bd889ed6d08286c11e2398c"
server_key_version: "encryptionkey@0"
I20260929 14:46:13.825426 20870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:13.830194 20870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:13.833259 20870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:13.833501 20870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:13.840284 20870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42021
I20260929 14:46:13.840314 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42021 every 8 connection(s)
I20260929 14:46:13.841271 20870 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:13.844203 20938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:13.845880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20870
I20260929 14:46:13.846032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:13.846324 392 external_mini_cluster.cc:1428] Setting key e7dbd4a7a2f453f521b9a01f4db97fbb
I20260929 14:46:13.850935 20938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485: Bootstrap starting.
I20260929 14:46:13.851794 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:13.847844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38303 (local address 127.0.0.1:42021)
0929 14:46:13.848281 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:13.848296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:13.848331 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:46:13.849147 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:13.849166 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:13.849472 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:13.849789 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:13.850566 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:13.850570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:13.850582 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:13.850593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:13.850650 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:13.850670 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:13.850799 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:13.850882 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 14:46:13.851042 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260929 14:46:13.853477 20938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:13.854324 20938 log.cc:824] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:13.854724 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:13.856369 20938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485: No bootstrap required, opened a new log
I20260929 14:46:13.858466 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:13.856697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50559 (local address 127.0.0.1:42021)
0929 14:46:13.856810 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:13.856814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:13.857002 (+ 188us) server_negotiation.cc:407] Connection header received
0929 14:46:13.857123 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:13.857126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:13.857160 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:13.857229 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:13.858035 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:13.858039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:13.858042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:13.858053 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:13.858080 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:13.858090 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:13.858163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:13.858240 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 14:46:13.858279 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:46:13.860451 20938 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } }
I20260929 14:46:13.860688 20938 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:13.860738 20938 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ae264007ab4a18911ac9c31b4be485, State: Initialized, Role: FOLLOWER
I20260929 14:46:13.861208 20938 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } }
I20260929 14:46:13.861340 20938 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:13.861382 20938 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:13.861477 20938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:13.862394 20938 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } }
I20260929 14:46:13.862727 20938 leader_election.cc:302] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ae264007ab4a18911ac9c31b4be485; no voters:
I20260929 14:46:13.863036 20938 leader_election.cc:288] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:13.863265 20948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:13.863472 20948 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [term 1 LEADER]: Becoming Leader. State: Replica: b5ae264007ab4a18911ac9c31b4be485, State: Running, Role: LEADER
I20260929 14:46:13.863791 20948 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } }
I20260929 14:46:13.864414 20938 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:13.865633 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ae264007ab4a18911ac9c31b4be485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } } }
I20260929 14:46:13.865665 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ae264007ab4a18911ac9c31b4be485. Latest consensus state: current_term: 1 leader_uuid: "b5ae264007ab4a18911ac9c31b4be485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ae264007ab4a18911ac9c31b4be485" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42021 } } }
I20260929 14:46:13.865814 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:13.866065 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:13.866228 20958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:13.869170 20958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:13.874571 20958 catalog_manager.cc:1347] Generated new cluster ID: 6df19b10291042e198b6b30a7b0825c5
I20260929 14:46:13.874640 20958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:13.887801 20958 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:13.888139 20958 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:13.895372 20958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ae264007ab4a18911ac9c31b4be485: Generated new TSK 0
I20260929 14:46:13.896136 20958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:13.901142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20870
2026-09-29T14:46:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:13 dist-test-slave-g8hp krb5kdc[20969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:13 dist-test-slave-g8hp krb5kdc[20969](info): set up 2 sockets
Sep 29 14:46:13 dist-test-slave-g8hp krb5kdc[20969](info): commencing operation
krb5kdc: starting...
W20260929 14:46:14.881001 392 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:14 dist-test-slave-g8hp krb5kdc[20969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:15.009727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:15.128192 20982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:15.128512 20982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:15.128598 20982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:15.133143 20982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:15.133234 20982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:15.133260 20982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:15.133281 20982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:15.133303 20982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:15.139294 20982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:15.140491 20982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:15.141630 20982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:15.150836 20987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:15.150944 20982 server_base.cc:1034] running on GCE node
W20260929 14:46:15.150863 20988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:15.150836 20990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:15.151937 20982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:15.153292 20982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:15.154490 20982 hybrid_clock.cc:648] HybridClock initialized: now 1790693175154461 us; error 46 us; skew 500 ppm
Sep 29 14:46:15 dist-test-slave-g8hp krb5kdc[20969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:15.157828 20982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:15.158948 20982 webserver.cc:460] Webserver started at http://127.0.0.1:41657/ using document root <none> and password file <none>
I20260929 14:46:15.159554 20982 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:15.159631 20982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:15.159843 20982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:15.162161 20982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "b5b6b4303ed1466b9336f96cadcf2ea6"
format_stamp: "Formatted at 2026-09-29 14:46:15 on dist-test-slave-g8hp"
server_key: "93fa3a703d653dedce27322231c69b8c"
server_key_iv: "cf267825deb6c8fb9f1b630da869b14a"
server_key_version: "encryptionkey@0"
I20260929 14:46:15.162760 20982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5b6b4303ed1466b9336f96cadcf2ea6"
format_stamp: "Formatted at 2026-09-29 14:46:15 on dist-test-slave-g8hp"
server_key: "93fa3a703d653dedce27322231c69b8c"
server_key_iv: "cf267825deb6c8fb9f1b630da869b14a"
server_key_version: "encryptionkey@0"
I20260929 14:46:15.166890 20982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 14:46:15.169600 20997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:15.171092 20982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:15.171249 20982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "b5b6b4303ed1466b9336f96cadcf2ea6"
format_stamp: "Formatted at 2026-09-29 14:46:15 on dist-test-slave-g8hp"
server_key: "93fa3a703d653dedce27322231c69b8c"
server_key_iv: "cf267825deb6c8fb9f1b630da869b14a"
server_key_version: "encryptionkey@0"
I20260929 14:46:15.171391 20982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:15.189338 20982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:15.192641 20982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:15.192824 20982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:15.199750 20982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38377
I20260929 14:46:15.199781 21049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38377 every 8 connection(s)
I20260929 14:46:15.200719 20982 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:15.203634 21050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:15.206904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 20982
I20260929 14:46:15.207032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:15.207278 392 external_mini_cluster.cc:1428] Setting key b9d0105a174f17c7e40d18081becb1a6
I20260929 14:46:15.209628 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6: Bootstrap starting.
I20260929 14:46:15.212522 21050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:15.213495 21050 log.cc:824] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:15.215854 21050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6: No bootstrap required, opened a new log
Sep 29 14:46:15 dist-test-slave-g8hp krb5kdc[20969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:46:15.220520 21050 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260929 14:46:15.220893 21050 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:15.220974 21050 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b6b4303ed1466b9336f96cadcf2ea6, State: Initialized, Role: FOLLOWER
I20260929 14:46:15.221515 21050 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260929 14:46:15.221701 21050 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:15.221779 21050 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:15.221889 21050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:15.223290 21050 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260929 14:46:15.223749 21050 leader_election.cc:302] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b6b4303ed1466b9336f96cadcf2ea6; no voters:
I20260929 14:46:15.224138 21050 leader_election.cc:288] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:15.224308 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:15.224527 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [term 1 LEADER]: Becoming Leader. State: Replica: b5b6b4303ed1466b9336f96cadcf2ea6, State: Running, Role: LEADER
I20260929 14:46:15.224792 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:15.208786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52051 (local address 127.0.0.1:38377)
0929 14:46:15.209273 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:15.209293 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:15.209335 (+ 42us) server_negotiation.cc:407] Connection header received
0929 14:46:15.210185 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:15.210210 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:15.210641 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:15.211034 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:15.211837 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:15.213502 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:15.214474 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:15.215189 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:15.215362 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:15.218327 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:15.218334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:15.218355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:15.218397 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:15.220746 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:15.221365 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:15.221372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:15.221379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:15.221563 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:15.222018 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:15.222022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:15.222024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:15.222625 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:15.222907 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:15.223254 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 14:46:15.223648 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":125,"threads_started":1}
I20260929 14:46:15.225042 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } }
I20260929 14:46:15.225569 21050 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:15.226811 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } } }
I20260929 14:46:15.226966 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:15.227497 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b6b4303ed1466b9336f96cadcf2ea6. Latest consensus state: current_term: 1 leader_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b6b4303ed1466b9336f96cadcf2ea6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38377 } } }
I20260929 14:46:15.227702 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:15.228142 21064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:15.228539 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:15.231644 21064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:15.238576 21064 catalog_manager.cc:1347] Generated new cluster ID: 031f80372f8949299ca0025b2cf67f80
I20260929 14:46:15.238655 21064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:15.238981 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:15.230672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60473 (local address 127.0.0.1:38377)
0929 14:46:15.230797 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:15.230801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:15.231028 (+ 227us) server_negotiation.cc:407] Connection header received
0929 14:46:15.231162 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:15.231165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:15.231209 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:15.231274 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:15.232116 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:15.232658 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:15.233524 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:15.233799 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:15.233889 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:15.235694 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:15.235698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:15.235700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:15.235746 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:15.237345 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:15.237749 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:15.237752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:15.237754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:15.237832 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:15.238082 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:15.238085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:15.238087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:15.238327 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:15.238441 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:15.238695 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 14:46:15.238838 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:46:15.252766 21064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:15.253891 21064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:15.271466 21064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b6b4303ed1466b9336f96cadcf2ea6: Generated new TSK 0
I20260929 14:46:15.272306 21064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:15.277053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 20982
2026-09-29T14:46:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:15 dist-test-slave-g8hp krb5kdc[21081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:15 dist-test-slave-g8hp krb5kdc[21081](info): set up 2 sockets
Sep 29 14:46:15 dist-test-slave-g8hp krb5kdc[21081](info): commencing operation
krb5kdc: starting...
W20260929 14:46:16.240908 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:16 dist-test-slave-g8hp krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:16.364645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:16.478889 21094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:16.479213 21094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:16.479257 21094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:16.483683 21094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:16.483790 21094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:16.483811 21094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:16.483829 21094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:16.483842 21094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:16.489694 21094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:16.490882 21094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:16.492002 21094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:16.501569 21099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:16.501686 21094 server_base.cc:1034] running on GCE node
W20260929 14:46:16.501561 21102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:16.501569 21100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:16.502542 21094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:16.503698 21094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:16.504902 21094 hybrid_clock.cc:648] HybridClock initialized: now 1790693176504850 us; error 34 us; skew 500 ppm
Sep 29 14:46:16 dist-test-slave-g8hp krb5kdc[21081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:16.515461 21094 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:16.516512 21094 webserver.cc:460] Webserver started at http://127.0.0.1:35603/ using document root <none> and password file <none>
I20260929 14:46:16.517125 21094 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:16.517201 21094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:16.517390 21094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:16.519537 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "9459ac9682374de1a7284dba644deb0e"
format_stamp: "Formatted at 2026-09-29 14:46:16 on dist-test-slave-g8hp"
server_key: "d863faf90762f4cf990c8276c31f4b96"
server_key_iv: "41157c1fc28dde9e0ce5902a7926b435"
server_key_version: "encryptionkey@0"
I20260929 14:46:16.520079 21094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "9459ac9682374de1a7284dba644deb0e"
format_stamp: "Formatted at 2026-09-29 14:46:16 on dist-test-slave-g8hp"
server_key: "d863faf90762f4cf990c8276c31f4b96"
server_key_iv: "41157c1fc28dde9e0ce5902a7926b435"
server_key_version: "encryptionkey@0"
I20260929 14:46:16.524031 21094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 14:46:16.526770 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:16.528234 21094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:16.528350 21094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "9459ac9682374de1a7284dba644deb0e"
format_stamp: "Formatted at 2026-09-29 14:46:16 on dist-test-slave-g8hp"
server_key: "d863faf90762f4cf990c8276c31f4b96"
server_key_iv: "41157c1fc28dde9e0ce5902a7926b435"
server_key_version: "encryptionkey@0"
I20260929 14:46:16.528448 21094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:16.557513 21094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:16.560806 21094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:16.561014 21094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:16.568050 21094 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43131
I20260929 14:46:16.568085 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43131 every 8 connection(s)
I20260929 14:46:16.569115 21094 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:16.571681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21094
I20260929 14:46:16.571823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:16.572085 392 external_mini_cluster.cc:1428] Setting key f249d0d32d48dee5b326a85ce93561bc
I20260929 14:46:16.572113 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:16.579632 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e: Bootstrap starting.
I20260929 14:46:16.581859 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:16.573682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51839 (local address 127.0.0.1:43131)
0929 14:46:16.574115 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:16.574129 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:16.574166 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:46:16.574988 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:16.575009 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:16.575379 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:16.575681 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:16.576081 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:16.577202 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:16.578430 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:16.578933 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:16.579048 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:16.579990 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:16.579995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:16.580010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:16.580445 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:16.580505 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:16.580529 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:16.580616 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:16.580756 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 14:46:16.581001 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260929 14:46:16.582121 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:16.582916 21162 log.cc:824] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:16.585202 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:16.585527 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e: No bootstrap required, opened a new log
I20260929 14:46:16.589357 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260929 14:46:16.589625 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:16.589704 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9459ac9682374de1a7284dba644deb0e, State: Initialized, Role: FOLLOWER
I20260929 14:46:16.590173 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260929 14:46:16.590332 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:16.590402 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:16.590507 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:16.590751 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:16.587311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32935 (local address 127.0.0.1:43131)
0929 14:46:16.587438 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:16.587443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:16.587457 (+ 14us) server_negotiation.cc:407] Connection header received
0929 14:46:16.587569 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:16.587572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:16.587627 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:16.587688 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:16.587956 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:16.588449 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:16.589290 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:16.589504 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:16.589600 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:16.590352 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:16.590356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:16.590359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:16.590371 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:16.590398 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:16.590410 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:16.590475 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:16.590568 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 14:46:16.590623 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:46:16.591634 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260929 14:46:16.591997 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9459ac9682374de1a7284dba644deb0e; no voters:
I20260929 14:46:16.592338 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:16.592438 21172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:16.592695 21172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [term 1 LEADER]: Becoming Leader. State: Replica: 9459ac9682374de1a7284dba644deb0e, State: Running, Role: LEADER
I20260929 14:46:16.593034 21172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } }
I20260929 14:46:16.593670 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:16.594702 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9459ac9682374de1a7284dba644deb0e. Latest consensus state: current_term: 1 leader_uuid: "9459ac9682374de1a7284dba644deb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } } }
I20260929 14:46:16.594826 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:16.595180 21181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:16.595250 21173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9459ac9682374de1a7284dba644deb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9459ac9682374de1a7284dba644deb0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43131 } } }
I20260929 14:46:16.595858 21173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:16.598436 21181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:16.604363 21181 catalog_manager.cc:1347] Generated new cluster ID: ac5597b4a742410c977aa6a5dc63e7ab
I20260929 14:46:16.604483 21181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:16.630582 21181 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:16.631706 21181 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:16.637092 21181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9459ac9682374de1a7284dba644deb0e: Generated new TSK 0
I20260929 14:46:16.637981 21181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:16.650265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21094
2026-09-29T14:46:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:16 dist-test-slave-g8hp krb5kdc[21193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:16 dist-test-slave-g8hp krb5kdc[21193](info): set up 2 sockets
Sep 29 14:46:16 dist-test-slave-g8hp krb5kdc[21193](info): commencing operation
krb5kdc: starting...
W20260929 14:46:17.625149 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:17 dist-test-slave-g8hp krb5kdc[21193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:17.752655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:17.867719 21206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:17.868036 21206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:17.868099 21206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:17.872308 21206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:17.872398 21206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:17.872426 21206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:17.872445 21206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:17.872465 21206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:17.878290 21206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:17.879454 21206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:17.880512 21206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:46:17.889631 21206 server_base.cc:1034] running on GCE node
W20260929 14:46:17.889592 21211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:17.889590 21212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:17.889590 21214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:17.890434 21206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:17.891601 21206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:17.892786 21206 hybrid_clock.cc:648] HybridClock initialized: now 1790693177892765 us; error 37 us; skew 500 ppm
Sep 29 14:46:17 dist-test-slave-g8hp krb5kdc[21193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:17.895884 21206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:17.896942 21206 webserver.cc:460] Webserver started at http://127.0.0.1:42143/ using document root <none> and password file <none>
I20260929 14:46:17.897512 21206 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:17.897593 21206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:17.897805 21206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:17.899976 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "5109b74363ca422db39d89024a1a552e"
format_stamp: "Formatted at 2026-09-29 14:46:17 on dist-test-slave-g8hp"
server_key: "d27ab2fa8410aced094a09672e93cd72"
server_key_iv: "d632637d4f6c2d63b4dd91fd9fcdb7cc"
server_key_version: "encryptionkey@0"
I20260929 14:46:17.900573 21206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "5109b74363ca422db39d89024a1a552e"
format_stamp: "Formatted at 2026-09-29 14:46:17 on dist-test-slave-g8hp"
server_key: "d27ab2fa8410aced094a09672e93cd72"
server_key_iv: "d632637d4f6c2d63b4dd91fd9fcdb7cc"
server_key_version: "encryptionkey@0"
I20260929 14:46:17.904510 21206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 14:46:17.907279 21221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:17.908742 21206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:17.908906 21206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "5109b74363ca422db39d89024a1a552e"
format_stamp: "Formatted at 2026-09-29 14:46:17 on dist-test-slave-g8hp"
server_key: "d27ab2fa8410aced094a09672e93cd72"
server_key_iv: "d632637d4f6c2d63b4dd91fd9fcdb7cc"
server_key_version: "encryptionkey@0"
I20260929 14:46:17.909039 21206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:17.913987 21206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:17.917315 21206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:17.917505 21206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:17.924713 21206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35233
I20260929 14:46:17.924714 21273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35233 every 8 connection(s)
I20260929 14:46:17.925849 21206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:17.929039 21274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:17.929448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21206
I20260929 14:46:17.929600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:17.929888 392 external_mini_cluster.cc:1428] Setting key f85098d0ae3a86c72360234d04b9e758
I20260929 14:46:17.935467 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:17.931335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33803 (local address 127.0.0.1:35233)
0929 14:46:17.931825 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:17.931844 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:17.931880 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:46:17.932720 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:17.932742 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:17.933109 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:17.933456 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:17.934272 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:17.934277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:17.934289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:17.934300 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:17.934359 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:17.934379 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:17.934508 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:17.934579 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 14:46:17.934737 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":161,"threads_started":1}
I20260929 14:46:17.936519 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e: Bootstrap starting.
I20260929 14:46:17.938330 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:17.938979 21274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:17.939756 21274 log.cc:824] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:17.941896 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:17.940422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51089 (local address 127.0.0.1:35233)
0929 14:46:17.940535 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:17.940538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:17.940611 (+ 73us) server_negotiation.cc:407] Connection header received
0929 14:46:17.940739 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:17.940742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:17.940777 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:17.940889 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:17.941558 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:17.941561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:17.941564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:17.941575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:17.941603 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:17.941615 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:17.941701 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:17.941748 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 14:46:17.941783 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 14:46:17.942246 21274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e: No bootstrap required, opened a new log
I20260929 14:46:17.946691 21274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } }
I20260929 14:46:17.947191 21274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:17.947273 21274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5109b74363ca422db39d89024a1a552e, State: Initialized, Role: FOLLOWER
I20260929 14:46:17.947762 21274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } }
I20260929 14:46:17.947906 21274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:17.947948 21274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:17.948027 21274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:17.949143 21274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } }
I20260929 14:46:17.949486 21274 leader_election.cc:302] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5109b74363ca422db39d89024a1a552e; no voters:
I20260929 14:46:17.949805 21274 leader_election.cc:288] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:17.949991 21284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:17.950357 21284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [term 1 LEADER]: Becoming Leader. State: Replica: 5109b74363ca422db39d89024a1a552e, State: Running, Role: LEADER
I20260929 14:46:17.950673 21284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } }
I20260929 14:46:17.951177 21274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:17.952364 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5109b74363ca422db39d89024a1a552e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } } }
I20260929 14:46:17.952522 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:17.952978 21293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:17.952930 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5109b74363ca422db39d89024a1a552e. Latest consensus state: current_term: 1 leader_uuid: "5109b74363ca422db39d89024a1a552e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5109b74363ca422db39d89024a1a552e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35233 } } }
I20260929 14:46:17.953092 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:17.956280 21293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:17.962795 21293 catalog_manager.cc:1347] Generated new cluster ID: 6491fde04b804d1ab1d989e0a9cdbac1
I20260929 14:46:17.962885 21293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:17.971889 21293 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:17.972275 21293 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:17.984256 21293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5109b74363ca422db39d89024a1a552e: Generated new TSK 0
I20260929 14:46:17.985234 21293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:17.991355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21206
2026-09-29T14:46:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:18 dist-test-slave-g8hp krb5kdc[21305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:18 dist-test-slave-g8hp krb5kdc[21305](info): set up 2 sockets
Sep 29 14:46:18 dist-test-slave-g8hp krb5kdc[21305](info): commencing operation
krb5kdc: starting...
W20260929 14:46:18.957122 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:19.087946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:19.203171 21318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:19.203490 21318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:19.203552 21318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:19.207778 21318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:19.207866 21318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:19.207893 21318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:19.207916 21318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:19.207937 21318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:19.213689 21318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:19.214851 21318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:19.215950 21318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:19.225174 21324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:19.225193 21323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:19.225346 21318 server_base.cc:1034] running on GCE node
W20260929 14:46:19.225203 21326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:19.226015 21318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:19.227329 21318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:19.228523 21318 hybrid_clock.cc:648] HybridClock initialized: now 1790693179228502 us; error 36 us; skew 500 ppm
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:19.232045 21318 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:19.233137 21318 webserver.cc:460] Webserver started at http://127.0.0.1:45673/ using document root <none> and password file <none>
I20260929 14:46:19.233764 21318 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:19.233851 21318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:19.234077 21318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:19.236420 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "e6684a0cd1774b78888194b1995c6c30"
format_stamp: "Formatted at 2026-09-29 14:46:19 on dist-test-slave-g8hp"
server_key: "7c04dd5d37a47774568338ffec8a63db"
server_key_iv: "d9b97a32ddc60a1ad21779131309614e"
server_key_version: "encryptionkey@0"
I20260929 14:46:19.237061 21318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6684a0cd1774b78888194b1995c6c30"
format_stamp: "Formatted at 2026-09-29 14:46:19 on dist-test-slave-g8hp"
server_key: "7c04dd5d37a47774568338ffec8a63db"
server_key_iv: "d9b97a32ddc60a1ad21779131309614e"
server_key_version: "encryptionkey@0"
I20260929 14:46:19.241101 21318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:19.243805 21333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:19.245563 21318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:19.245709 21318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "e6684a0cd1774b78888194b1995c6c30"
format_stamp: "Formatted at 2026-09-29 14:46:19 on dist-test-slave-g8hp"
server_key: "7c04dd5d37a47774568338ffec8a63db"
server_key_iv: "d9b97a32ddc60a1ad21779131309614e"
server_key_version: "encryptionkey@0"
I20260929 14:46:19.245834 21318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:19.259603 21318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:19.263065 21318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:19.263279 21318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:19.271279 21318 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36241
I20260929 14:46:19.271320 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36241 every 8 connection(s)
I20260929 14:46:19.272298 21318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:19.274824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21318
I20260929 14:46:19.274963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:19.275271 392 external_mini_cluster.cc:1428] Setting key 562ef7771d8e5d5e7ca912d5c6a049f1
I20260929 14:46:19.275517 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:19.282933 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: Bootstrap starting.
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:46:19.285718 21386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:19.286453 21386 log.cc:824] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:19.288720 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: No bootstrap required, opened a new log
I20260929 14:46:19.290717 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:19.276694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:43937 (local address 127.0.0.1:36241)
0929 14:46:19.277188 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:19.277208 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:19.277244 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:46:19.277918 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:19.277936 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:19.278315 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:19.278613 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:19.279562 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:19.280583 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:19.281799 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:19.282284 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:19.282400 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:19.285420 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:19.285426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:19.285440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:19.285473 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:19.287466 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:19.288128 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:19.288133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:19.288137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:19.288249 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:19.288595 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:19.288597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:19.288598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:19.289158 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:19.289323 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:19.289693 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 14:46:19.289951 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":183,"threads_started":1}
I20260929 14:46:19.292832 21386 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260929 14:46:19.293148 21386 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:19.293196 21386 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6684a0cd1774b78888194b1995c6c30, State: Initialized, Role: FOLLOWER
I20260929 14:46:19.293632 21386 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260929 14:46:19.293788 21386 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:19.293861 21386 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:19.293962 21386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:19.294068 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:19.295176 21386 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260929 14:46:19.295588 21386 leader_election.cc:302] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6684a0cd1774b78888194b1995c6c30; no voters:
I20260929 14:46:19.295987 21386 leader_election.cc:288] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:19.296192 21395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:19.296434 21395 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [term 1 LEADER]: Becoming Leader. State: Replica: e6684a0cd1774b78888194b1995c6c30, State: Running, Role: LEADER
I20260929 14:46:19.297000 21395 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } }
I20260929 14:46:19.297500 21386 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:19.298592 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6684a0cd1774b78888194b1995c6c30. Latest consensus state: current_term: 1 leader_uuid: "e6684a0cd1774b78888194b1995c6c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } } }
I20260929 14:46:19.298729 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:19.299057 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6684a0cd1774b78888194b1995c6c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6684a0cd1774b78888194b1995c6c30" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36241 } } }
I20260929 14:46:19.299145 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30 [sys.catalog]: This master's current role is: LEADER
W20260929 14:46:19.302947 21411 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:19.303107 21411 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:19.303184 21405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:19.304141 21405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:19.306356 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:19.296547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:45883 (local address 127.0.0.1:36241)
0929 14:46:19.296659 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:19.296664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:19.296933 (+ 269us) server_negotiation.cc:407] Connection header received
0929 14:46:19.297125 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:19.297129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:19.297177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:19.297224 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:19.298191 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:19.298989 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:19.300323 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:19.300575 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:19.300659 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:19.302380 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:19.302384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:19.302387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:19.302416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:19.304160 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:19.304762 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:19.304765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:19.304767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:19.304853 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:19.305274 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:19.305278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:19.305280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:19.305511 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:19.305634 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:19.306096 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 14:46:19.306197 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 14:46:19.310717 21405 catalog_manager.cc:1347] Generated new cluster ID: 38a4880d4565474faf159a6e3fc30db8
I20260929 14:46:19.310803 21405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:19.317667 21405 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:19.319034 21405 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:19.329331 21405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6684a0cd1774b78888194b1995c6c30: Generated new TSK 0
I20260929 14:46:19.330307 21405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:19.343711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21318
W20260929 14:46:19.348706 21303 connection.cc:535] client connection to 127.0.0.1:36241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36241): Connection reset by peer (error 104)
2026-09-29T14:46:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21417](info): set up 2 sockets
Sep 29 14:46:19 dist-test-slave-g8hp krb5kdc[21417](info): commencing operation
krb5kdc: starting...
W20260929 14:46:20.301250 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:20 dist-test-slave-g8hp krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:20.429275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:20.540122 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:20.540407 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:20.540450 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:20.544471 21430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:20.544553 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:20.544584 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:20.544605 21430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:20.544622 21430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:20.550000 21430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:20.551028 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:20.551990 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:20.560634 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:20.560679 21435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:20.560731 21430 server_base.cc:1034] running on GCE node
W20260929 14:46:20.560678 21436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:20.561546 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:20.562747 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:20.564011 21430 hybrid_clock.cc:648] HybridClock initialized: now 1790693180563899 us; error 48 us; skew 500 ppm
Sep 29 14:46:20 dist-test-slave-g8hp krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:20.566828 21430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:20.567880 21430 webserver.cc:460] Webserver started at http://127.0.0.1:46363/ using document root <none> and password file <none>
I20260929 14:46:20.568424 21430 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:20.568492 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:20.568702 21430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:20.570781 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "a51de71471014f1f85002af2f2620923"
format_stamp: "Formatted at 2026-09-29 14:46:20 on dist-test-slave-g8hp"
server_key: "b290e15cf8df437c88a8d58ddfe5fb49"
server_key_iv: "292e68757b33359d395b0942c6cfbc54"
server_key_version: "encryptionkey@0"
I20260929 14:46:20.571339 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "a51de71471014f1f85002af2f2620923"
format_stamp: "Formatted at 2026-09-29 14:46:20 on dist-test-slave-g8hp"
server_key: "b290e15cf8df437c88a8d58ddfe5fb49"
server_key_iv: "292e68757b33359d395b0942c6cfbc54"
server_key_version: "encryptionkey@0"
I20260929 14:46:20.575246 21430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:46:20.577957 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:20.579360 21430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 14:46:20.579545 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "a51de71471014f1f85002af2f2620923"
format_stamp: "Formatted at 2026-09-29 14:46:20 on dist-test-slave-g8hp"
server_key: "b290e15cf8df437c88a8d58ddfe5fb49"
server_key_iv: "292e68757b33359d395b0942c6cfbc54"
server_key_version: "encryptionkey@0"
I20260929 14:46:20.579658 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:20.607095 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:20.610421 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:20.610622 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:20.617913 21430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44349
I20260929 14:46:20.617924 21497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44349 every 8 connection(s)
I20260929 14:46:20.618919 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:20.621812 21498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:20.626099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21430
I20260929 14:46:20.626248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:20.626497 392 external_mini_cluster.cc:1428] Setting key 98bacb76d2f56956a282ffa7f5cfd163
I20260929 14:46:20.627893 21498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: Bootstrap starting.
I20260929 14:46:20.630330 21498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:20.631172 21498 log.cc:824] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:20.633364 21498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: No bootstrap required, opened a new log
I20260929 14:46:20.635516 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:20.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:41327 (local address 127.0.0.1:44349)
0929 14:46:20.628293 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:20.628309 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:20.628344 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:46:20.629075 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:20.629097 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:20.629456 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:20.629752 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:20.630103 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:20.631244 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:20.632264 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:20.632743 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:20.632890 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:20.633834 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:20.633839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:20.633852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:20.634256 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:20.634316 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:20.634339 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:20.634455 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:20.634642 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:46:20.634772 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":138,"threads_started":1}
I20260929 14:46:20.637079 21498 raft_consensus.cc:348] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260929 14:46:20.637354 21498 raft_consensus.cc:374] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:20.637421 21498 raft_consensus.cc:729] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51de71471014f1f85002af2f2620923, State: Initialized, Role: FOLLOWER
I20260929 14:46:20.637857 21498 consensus_queue.cc:260] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260929 14:46:20.637991 21498 raft_consensus.cc:388] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:20.638052 21498 raft_consensus.cc:482] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:20.638147 21498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:20.638561 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:20.639288 21498 raft_consensus.cc:504] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260929 14:46:20.639663 21498 leader_election.cc:302] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a51de71471014f1f85002af2f2620923; no voters:
I20260929 14:46:20.639999 21498 leader_election.cc:288] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:20.640086 21506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:20.640362 21506 raft_consensus.cc:686] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [term 1 LEADER]: Becoming Leader. State: Replica: a51de71471014f1f85002af2f2620923, State: Running, Role: LEADER
I20260929 14:46:20.640743 21506 consensus_queue.cc:237] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } }
I20260929 14:46:20.641469 21498 sys_catalog.cc:564] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:20.644351 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a51de71471014f1f85002af2f2620923. Latest consensus state: current_term: 1 leader_uuid: "a51de71471014f1f85002af2f2620923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260929 14:46:20.644490 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:20.644410 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a51de71471014f1f85002af2f2620923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a51de71471014f1f85002af2f2620923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44349 } } }
I20260929 14:46:20.644544 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:20.645676 21501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:20.640775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:39173 (local address 127.0.0.1:44349)
0929 14:46:20.640907 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:20.640912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:20.641165 (+ 253us) server_negotiation.cc:407] Connection header received
0929 14:46:20.641299 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:20.641302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:20.641352 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:20.641430 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:20.641859 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:20.642624 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:20.643638 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:20.643963 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:20.644077 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:20.645198 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:20.645200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:20.645203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:20.645215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:20.645239 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:20.645249 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:20.645319 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:20.645504 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 14:46:20.645544 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 14:46:20.646970 21522 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:20.647035 21522 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:20.647106 21524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:20.647787 21524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:20.653357 21524 catalog_manager.cc:1347] Generated new cluster ID: cca739af31bc4b66829b443763abca55
I20260929 14:46:20.653429 21524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:20.672232 21524 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:20.673309 21524 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:20.685168 21524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a51de71471014f1f85002af2f2620923: Generated new TSK 0
I20260929 14:46:20.685925 21524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:20.692602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21430
W20260929 14:46:20.697500 21415 connection.cc:535] client connection to 127.0.0.1:44349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44349): Connection reset by peer (error 104)
2026-09-29T14:46:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:20 dist-test-slave-g8hp krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:20 dist-test-slave-g8hp krb5kdc[21529](info): set up 2 sockets
Sep 29 14:46:20 dist-test-slave-g8hp krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260929 14:46:21.656975 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:21 dist-test-slave-g8hp krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:21.784574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:21.897645 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:21.897953 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:21.897996 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:21.902184 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:21.902264 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:21.902282 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:21.902297 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:21.902312 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:21.908056 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:21.909246 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:21.910318 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:21.919546 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:21.919520 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:21.919525 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:21.920094 21542 server_base.cc:1034] running on GCE node
I20260929 14:46:21.920711 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:21.921936 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:21.923110 21542 hybrid_clock.cc:648] HybridClock initialized: now 1790693181923082 us; error 43 us; skew 500 ppm
Sep 29 14:46:21 dist-test-slave-g8hp krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:21.931828 21542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:21.933019 21542 webserver.cc:460] Webserver started at http://127.0.0.1:33633/ using document root <none> and password file <none>
I20260929 14:46:21.933669 21542 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:21.933744 21542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:21.933964 21542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:21.936122 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "21306fb883ac4d7dab1c53e340e953ed"
format_stamp: "Formatted at 2026-09-29 14:46:21 on dist-test-slave-g8hp"
server_key: "80bf898eb459b60c7160fb86b4268124"
server_key_iv: "d32cc4c1365a5b7ca9f4c95f3d5c8734"
server_key_version: "encryptionkey@0"
I20260929 14:46:21.936677 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "21306fb883ac4d7dab1c53e340e953ed"
format_stamp: "Formatted at 2026-09-29 14:46:21 on dist-test-slave-g8hp"
server_key: "80bf898eb459b60c7160fb86b4268124"
server_key_iv: "d32cc4c1365a5b7ca9f4c95f3d5c8734"
server_key_version: "encryptionkey@0"
I20260929 14:46:21.940636 21542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 14:46:21.943194 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:21.944605 21542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 14:46:21.944748 21542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "21306fb883ac4d7dab1c53e340e953ed"
format_stamp: "Formatted at 2026-09-29 14:46:21 on dist-test-slave-g8hp"
server_key: "80bf898eb459b60c7160fb86b4268124"
server_key_iv: "d32cc4c1365a5b7ca9f4c95f3d5c8734"
server_key_version: "encryptionkey@0"
I20260929 14:46:21.944900 21542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:21.949820 21542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:21.953290 21542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:21.953485 21542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:21.960296 21542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37467
I20260929 14:46:21.960307 21609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37467 every 8 connection(s)
I20260929 14:46:21.961292 21542 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:21.964247 21610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:21.970259 21610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed: Bootstrap starting.
I20260929 14:46:21.971153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21542
I20260929 14:46:21.971285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:21.971572 392 external_mini_cluster.cc:1428] Setting key aa95a3a49e739c265b4ad1ac9e0cab0e
I20260929 14:46:21.973305 21610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:21.974061 21610 log.cc:824] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:21.976260 21610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed: No bootstrap required, opened a new log
I20260929 14:46:21.976797 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:21.972759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:34417 (local address 127.0.0.1:37467)
0929 14:46:21.973241 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:21.973256 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:21.973292 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:46:21.974006 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:21.974031 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:21.974361 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:21.974749 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:21.975473 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:21.975477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:21.975489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:21.975498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:21.975545 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:21.975564 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:21.975647 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:21.975764 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 14:46:21.976046 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":145,"threads_started":1}
I20260929 14:46:21.979630 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:21.980124 21610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260929 14:46:21.980441 21610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:21.980510 21610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21306fb883ac4d7dab1c53e340e953ed, State: Initialized, Role: FOLLOWER
I20260929 14:46:21.981037 21610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260929 14:46:21.981220 21610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:21.981292 21610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:21.981392 21610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:21.982568 21610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260929 14:46:21.982952 21610 leader_election.cc:302] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21306fb883ac4d7dab1c53e340e953ed; no voters:
I20260929 14:46:21.983280 21610 leader_election.cc:288] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:21.983429 21620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:21.983676 21620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [term 1 LEADER]: Becoming Leader. State: Replica: 21306fb883ac4d7dab1c53e340e953ed, State: Running, Role: LEADER
I20260929 14:46:21.983824 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:21.982006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:42659 (local address 127.0.0.1:37467)
0929 14:46:21.982131 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:21.982135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:21.982378 (+ 243us) server_negotiation.cc:407] Connection header received
0929 14:46:21.982519 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:21.982521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:21.982558 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:21.982622 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:21.983372 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:21.983376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:21.983379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:21.983391 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:21.983423 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:21.983437 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:21.983532 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:21.983663 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 14:46:21.983707 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:46:21.984172 21620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } }
I20260929 14:46:21.984592 21610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:21.985522 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21306fb883ac4d7dab1c53e340e953ed. Latest consensus state: current_term: 1 leader_uuid: "21306fb883ac4d7dab1c53e340e953ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260929 14:46:21.985693 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:21.985908 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21306fb883ac4d7dab1c53e340e953ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21306fb883ac4d7dab1c53e340e953ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37467 } } }
I20260929 14:46:21.986016 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:21.986351 21629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:21.989152 21629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:21.994503 21629 catalog_manager.cc:1347] Generated new cluster ID: 801c0f30ecad43a4a3ad5807d5aa7837
I20260929 14:46:21.994628 21629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:22.006620 21629 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:22.007030 21629 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:22.010772 21629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21306fb883ac4d7dab1c53e340e953ed: Generated new TSK 0
I20260929 14:46:22.011632 21629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:22.017163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21542
2026-09-29T14:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:22 dist-test-slave-g8hp krb5kdc[21641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:22 dist-test-slave-g8hp krb5kdc[21641](info): set up 2 sockets
Sep 29 14:46:22 dist-test-slave-g8hp krb5kdc[21641](info): commencing operation
krb5kdc: starting...
W20260929 14:46:22.977012 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:23.105202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:23.215471 21654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:23.215755 21654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:23.215798 21654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:23.219843 21654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:23.219914 21654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:23.219933 21654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:23.219949 21654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:23.219962 21654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:23.225390 21654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:23.226439 21654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:23.227393 21654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:23.236128 21660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:23.236128 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:23.236299 21654 server_base.cc:1034] running on GCE node
W20260929 14:46:23.236128 21662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:23.237080 21654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:23.238294 21654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:23.239462 21654 hybrid_clock.cc:648] HybridClock initialized: now 1790693183239442 us; error 37 us; skew 500 ppm
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:23.242316 21654 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:23.243258 21654 webserver.cc:460] Webserver started at http://127.0.0.1:44841/ using document root <none> and password file <none>
I20260929 14:46:23.243882 21654 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:23.243953 21654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:23.244150 21654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:23.246488 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "523e87cf8107417b87a28e212bcf11b9"
format_stamp: "Formatted at 2026-09-29 14:46:23 on dist-test-slave-g8hp"
server_key: "93f8d6acb7da93e400bb606a4aae21e4"
server_key_iv: "cffcfc6df04ba5bc7b7380372ee92191"
server_key_version: "encryptionkey@0"
I20260929 14:46:23.247053 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "523e87cf8107417b87a28e212bcf11b9"
format_stamp: "Formatted at 2026-09-29 14:46:23 on dist-test-slave-g8hp"
server_key: "93f8d6acb7da93e400bb606a4aae21e4"
server_key_iv: "cffcfc6df04ba5bc7b7380372ee92191"
server_key_version: "encryptionkey@0"
I20260929 14:46:23.251111 21654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:46:23.253813 21669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:23.255194 21654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 14:46:23.255326 21654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "523e87cf8107417b87a28e212bcf11b9"
format_stamp: "Formatted at 2026-09-29 14:46:23 on dist-test-slave-g8hp"
server_key: "93f8d6acb7da93e400bb606a4aae21e4"
server_key_iv: "cffcfc6df04ba5bc7b7380372ee92191"
server_key_version: "encryptionkey@0"
I20260929 14:46:23.255443 21654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:23.286032 21654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:23.289520 21654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:23.289733 21654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:23.297017 21654 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39565
I20260929 14:46:23.297035 21721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39565 every 8 connection(s)
I20260929 14:46:23.298058 21654 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:23.302011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21654
I20260929 14:46:23.302183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:23.302444 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:23.302466 392 external_mini_cluster.cc:1428] Setting key b9d2fc869df0b9ce2a914a4060840bce
I20260929 14:46:23.309274 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: Bootstrap starting.
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 14:46:23.311975 21722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:23.312887 21722 log.cc:824] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:23.315228 21722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: No bootstrap required, opened a new log
I20260929 14:46:23.317384 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:23.303902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:40421 (local address 127.0.0.1:39565)
0929 14:46:23.304366 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:23.304386 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:23.304583 (+ 197us) server_negotiation.cc:407] Connection header received
0929 14:46:23.305367 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:23.305386 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:23.305765 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:23.306103 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:23.306928 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:23.307938 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:23.308805 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:23.309328 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:23.309492 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:23.312459 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:23.312465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:23.312480 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:23.312510 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:23.314238 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:23.314852 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:23.314857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:23.314862 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:23.314996 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:23.315367 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:23.315371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:23.315372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:23.315783 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:23.316011 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:23.316277 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 14:46:23.316485 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":142,"threads_started":1}
I20260929 14:46:23.319051 21722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } }
I20260929 14:46:23.319339 21722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:23.319410 21722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523e87cf8107417b87a28e212bcf11b9, State: Initialized, Role: FOLLOWER
I20260929 14:46:23.319823 21722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } }
I20260929 14:46:23.319962 21722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:23.320025 21722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:23.320123 21722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:23.320304 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:23.321333 21722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } }
I20260929 14:46:23.321738 21722 leader_election.cc:302] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523e87cf8107417b87a28e212bcf11b9; no voters:
I20260929 14:46:23.322081 21722 leader_election.cc:288] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:23.322680 21731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:23.322938 21731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [term 1 LEADER]: Becoming Leader. State: Replica: 523e87cf8107417b87a28e212bcf11b9, State: Running, Role: LEADER
I20260929 14:46:23.323288 21731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } }
I20260929 14:46:23.323491 21722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:23.325045 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523e87cf8107417b87a28e212bcf11b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } } }
I20260929 14:46:23.325171 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:23.325527 21734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523e87cf8107417b87a28e212bcf11b9. Latest consensus state: current_term: 1 leader_uuid: "523e87cf8107417b87a28e212bcf11b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523e87cf8107417b87a28e212bcf11b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39565 } } }
I20260929 14:46:23.326427 21734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9 [sys.catalog]: This master's current role is: LEADER
W20260929 14:46:23.328325 21747 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:23.328397 21747 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:23.328469 21746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:23.329533 21746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:23.332513 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:23.322498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:40599 (local address 127.0.0.1:39565)
0929 14:46:23.322611 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:23.322616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:23.322881 (+ 265us) server_negotiation.cc:407] Connection header received
0929 14:46:23.323042 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:23.323045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:23.323095 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:23.323144 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:23.324198 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:23.325180 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:23.326302 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:23.326625 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:23.326721 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:23.328670 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:23.328673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:23.328676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:23.328704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:23.330516 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:23.331171 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:23.331174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:23.331176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:23.331269 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:23.331653 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:23.331656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:23.331658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:23.331856 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:23.331986 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:23.332271 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 14:46:23.332367 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:46:23.336361 21746 catalog_manager.cc:1347] Generated new cluster ID: ba4fa895e2214db2a066570d784d587b
I20260929 14:46:23.336485 21746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:23.346750 21746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:23.348001 21746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:23.358377 21746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523e87cf8107417b87a28e212bcf11b9: Generated new TSK 0
I20260929 14:46:23.359143 21746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:23.371752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21654
W20260929 14:46:23.376631 21639 connection.cc:535] client connection to 127.0.0.1:39565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39565): Connection reset by peer (error 104)
2026-09-29T14:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21753](info): set up 2 sockets
Sep 29 14:46:23 dist-test-slave-g8hp krb5kdc[21753](info): commencing operation
krb5kdc: starting...
W20260929 14:46:24.333010 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:24 dist-test-slave-g8hp krb5kdc[21753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:24.459566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:24.570552 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:24.570883 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:24.570946 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:24.575141 21766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:24.575222 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:24.575250 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:24.575273 21766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:24.575299 21766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:24.580832 21766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:24.582017 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:24.583081 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:24.592461 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:24.592541 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:24.592519 21766 server_base.cc:1034] running on GCE node
W20260929 14:46:24.592476 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:24.593323 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:24.594511 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:24.595679 21766 hybrid_clock.cc:648] HybridClock initialized: now 1790693184595657 us; error 38 us; skew 500 ppm
Sep 29 14:46:24 dist-test-slave-g8hp krb5kdc[21753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:24.606545 21766 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:24.607568 21766 webserver.cc:460] Webserver started at http://127.0.0.1:33677/ using document root <none> and password file <none>
I20260929 14:46:24.608125 21766 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:24.608186 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:24.608358 21766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:24.610532 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "39530ba31f944ff687d4eb0449c73204"
format_stamp: "Formatted at 2026-09-29 14:46:24 on dist-test-slave-g8hp"
server_key: "56692933514270cba4ba4b50f2801393"
server_key_iv: "69dc3689df4be352670dd380921e178b"
server_key_version: "encryptionkey@0"
I20260929 14:46:24.611083 21766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "39530ba31f944ff687d4eb0449c73204"
format_stamp: "Formatted at 2026-09-29 14:46:24 on dist-test-slave-g8hp"
server_key: "56692933514270cba4ba4b50f2801393"
server_key_iv: "69dc3689df4be352670dd380921e178b"
server_key_version: "encryptionkey@0"
I20260929 14:46:24.615010 21766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:24.617635 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:24.618945 21766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:24.619072 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "39530ba31f944ff687d4eb0449c73204"
format_stamp: "Formatted at 2026-09-29 14:46:24 on dist-test-slave-g8hp"
server_key: "56692933514270cba4ba4b50f2801393"
server_key_iv: "69dc3689df4be352670dd380921e178b"
server_key_version: "encryptionkey@0"
I20260929 14:46:24.619180 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:24.654767 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:24.657908 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:24.658077 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:24.664814 21766 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40457
I20260929 14:46:24.664832 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40457 every 8 connection(s)
I20260929 14:46:24.665807 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:24.666561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21766
I20260929 14:46:24.666702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:24.666982 392 external_mini_cluster.cc:1428] Setting key 7c4303197b685ae18e90617ad8aa39b9
I20260929 14:46:24.669102 21834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:24.675554 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: Bootstrap starting.
I20260929 14:46:24.676142 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:24.668286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:36331 (local address 127.0.0.1:40457)
0929 14:46:24.668719 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:24.668734 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:24.668770 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:46:24.669494 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:24.669514 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:24.669867 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:24.670209 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:24.670537 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:24.671532 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:24.672748 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:24.673289 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:24.673407 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:24.674335 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:24.674340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:24.674353 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:24.674741 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:24.674798 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:24.674821 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:24.674929 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:24.675116 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 14:46:24.675257 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":145,"threads_started":1}
I20260929 14:46:24.677963 21834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:24.678750 21834 log.cc:824] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:24.680781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:24.684166 21834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: No bootstrap required, opened a new log
I20260929 14:46:24.689781 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:24.683250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:49239 (local address 127.0.0.1:40457)
0929 14:46:24.683375 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:24.683380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:24.685160 (+ 1780us) server_negotiation.cc:407] Connection header received
0929 14:46:24.685333 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:24.685337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:24.685397 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:24.685482 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:24.685817 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:24.686562 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:24.687717 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:24.688048 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:24.688155 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:24.689144 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:24.689147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:24.689149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:24.689183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:24.689215 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:24.689230 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:24.689326 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:24.689535 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 14:46:24.689592 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 14:46:24.689634 21834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260929 14:46:24.690207 21834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:24.690311 21834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39530ba31f944ff687d4eb0449c73204, State: Initialized, Role: FOLLOWER
I20260929 14:46:24.690917 21834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260929 14:46:24.691242 21834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:24.691342 21834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:24.691468 21834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:24.692972 21834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260929 14:46:24.693475 21834 leader_election.cc:302] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39530ba31f944ff687d4eb0449c73204; no voters:
I20260929 14:46:24.693889 21834 leader_election.cc:288] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:24.694005 21844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:24.694284 21844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [term 1 LEADER]: Becoming Leader. State: Replica: 39530ba31f944ff687d4eb0449c73204, State: Running, Role: LEADER
I20260929 14:46:24.694752 21844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260929 14:46:24.695632 21834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:24.696461 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39530ba31f944ff687d4eb0449c73204. Latest consensus state: current_term: 1 leader_uuid: "39530ba31f944ff687d4eb0449c73204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260929 14:46:24.696592 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:24.696942 21846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39530ba31f944ff687d4eb0449c73204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39530ba31f944ff687d4eb0449c73204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260929 14:46:24.697058 21846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204 [sys.catalog]: This master's current role is: LEADER
W20260929 14:46:24.700485 21859 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:24.700696 21859 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:24.700783 21853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:24.701766 21853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:24.708767 21853 catalog_manager.cc:1347] Generated new cluster ID: c3265b1acaa848f2b9cd682bc3882e58
I20260929 14:46:24.708889 21853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:24.724853 21853 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:24.726205 21853 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:24.734015 21853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39530ba31f944ff687d4eb0449c73204: Generated new TSK 0
I20260929 14:46:24.734921 21853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:24.740109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21766
W20260929 14:46:24.746304 21751 connection.cc:535] client connection to 127.0.0.1:40457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40457): Connection reset by peer (error 104)
2026-09-29T14:46:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:24 dist-test-slave-g8hp krb5kdc[21865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:24 dist-test-slave-g8hp krb5kdc[21865](info): set up 2 sockets
Sep 29 14:46:24 dist-test-slave-g8hp krb5kdc[21865](info): commencing operation
krb5kdc: starting...
W20260929 14:46:25.700994 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:25 dist-test-slave-g8hp krb5kdc[21865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 14:46:25.829312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:25.942082 21878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:25.942394 21878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:25.942456 21878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:25.946698 21878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:25.946790 21878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:25.946811 21878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:25.946830 21878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:25.946843 21878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:25.952571 21878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:25.953760 21878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:25.954926 21878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:25.964133 21883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:25.964129 21884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:25.964448 21878 server_base.cc:1034] running on GCE node
W20260929 14:46:25.964468 21886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:25.965092 21878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:25.966279 21878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:25.967484 21878 hybrid_clock.cc:648] HybridClock initialized: now 1790693185967466 us; error 31 us; skew 500 ppm
Sep 29 14:46:25 dist-test-slave-g8hp krb5kdc[21865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:25.970532 21878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 14:46:25.971614 21878 webserver.cc:460] Webserver started at http://127.0.0.1:34973/ using document root <none> and password file <none>
I20260929 14:46:25.972169 21878 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:25.972236 21878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:25.972441 21878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:25.974612 21878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "88312eebf29b43bf866aef162df8b08b"
format_stamp: "Formatted at 2026-09-29 14:46:25 on dist-test-slave-g8hp"
server_key: "ff3174e9968fcef3c07a85b0369a69b9"
server_key_iv: "5a98714952ec8f8726cc093662e10ee2"
server_key_version: "encryptionkey@0"
I20260929 14:46:25.975167 21878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "88312eebf29b43bf866aef162df8b08b"
format_stamp: "Formatted at 2026-09-29 14:46:25 on dist-test-slave-g8hp"
server_key: "ff3174e9968fcef3c07a85b0369a69b9"
server_key_iv: "5a98714952ec8f8726cc093662e10ee2"
server_key_version: "encryptionkey@0"
I20260929 14:46:25.979362 21878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 14:46:25.982146 21893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:25.983732 21878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:46:25.983912 21878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "88312eebf29b43bf866aef162df8b08b"
format_stamp: "Formatted at 2026-09-29 14:46:25 on dist-test-slave-g8hp"
server_key: "ff3174e9968fcef3c07a85b0369a69b9"
server_key_iv: "5a98714952ec8f8726cc093662e10ee2"
server_key_version: "encryptionkey@0"
I20260929 14:46:25.984040 21878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:25.988991 21878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:25.992515 21878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:25.992715 21878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:25.999809 21878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46809
I20260929 14:46:25.999846 21945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46809 every 8 connection(s)
I20260929 14:46:26.001107 21878 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:26.004156 21946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:26.005647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21878
I20260929 14:46:26.005800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:26.006099 392 external_mini_cluster.cc:1428] Setting key d51b5ec3bca5e4d9ea50af9a1cb04393
I20260929 14:46:26.011005 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b: Bootstrap starting.
I20260929 14:46:26.011693 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:26.007545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:57363 (local address 127.0.0.1:46809)
0929 14:46:26.008001 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:26.008024 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:26.008061 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:46:26.008715 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:26.008733 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:26.009100 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:26.009422 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:26.010272 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:26.010277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:26.010290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:26.010300 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:26.010355 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:26.010377 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:26.010501 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:26.010593 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 14:46:26.010774 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260929 14:46:26.013484 21946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:26.014439 21946 log.cc:824] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:26.014680 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:26.017472 21946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b: No bootstrap required, opened a new log
I20260929 14:46:26.019348 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:26.017714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.107:51667 (local address 127.0.0.1:46809)
0929 14:46:26.017834 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:26.017838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:26.018014 (+ 176us) server_negotiation.cc:407] Connection header received
0929 14:46:26.018140 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:26.018143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:26.018180 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:26.018244 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:26.018957 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:26.018961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:26.018964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:26.018976 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:26.019005 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:26.019017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:26.019093 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:26.019169 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 14:46:26.019212 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 14:46:26.021603 21946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } }
I20260929 14:46:26.021886 21946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:26.021924 21946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88312eebf29b43bf866aef162df8b08b, State: Initialized, Role: FOLLOWER
I20260929 14:46:26.022342 21946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } }
I20260929 14:46:26.022488 21946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:26.022539 21946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:26.022619 21946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:26.023648 21946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } }
I20260929 14:46:26.024062 21946 leader_election.cc:302] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88312eebf29b43bf866aef162df8b08b; no voters:
I20260929 14:46:26.024435 21946 leader_election.cc:288] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:26.024607 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:26.024842 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [term 1 LEADER]: Becoming Leader. State: Replica: 88312eebf29b43bf866aef162df8b08b, State: Running, Role: LEADER
I20260929 14:46:26.025367 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } }
I20260929 14:46:26.025954 21946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:26.027295 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88312eebf29b43bf866aef162df8b08b. Latest consensus state: current_term: 1 leader_uuid: "88312eebf29b43bf866aef162df8b08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } } }
I20260929 14:46:26.027444 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:26.027274 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88312eebf29b43bf866aef162df8b08b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88312eebf29b43bf866aef162df8b08b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46809 } } }
I20260929 14:46:26.027683 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:26.027972 21968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:26.031386 21968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:26.037914 21968 catalog_manager.cc:1347] Generated new cluster ID: 5af86ae74f5b4b71b8c274591e91592a
I20260929 14:46:26.038007 21968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:26.048192 21968 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:26.048544 21968 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:26.058868 21968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88312eebf29b43bf866aef162df8b08b: Generated new TSK 0
I20260929 14:46:26.059597 21968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:26.069986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21878
2026-09-29T14:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:26 dist-test-slave-g8hp krb5kdc[21977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:26 dist-test-slave-g8hp krb5kdc[21977](info): set up 2 sockets
Sep 29 14:46:26 dist-test-slave-g8hp krb5kdc[21977](info): commencing operation
krb5kdc: starting...
W20260929 14:46:27.036901 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:27.164480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:27.274907 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:27.275238 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:27.275302 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:27.280041 21990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:27.280138 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:27.280169 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:27.280192 21990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:27.280212 21990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:27.286036 21990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:27.287199 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:27.288262 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:27.297070 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:27.297078 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:27.297123 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:27.297232 21990 server_base.cc:1034] running on GCE node
I20260929 14:46:27.297987 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:27.299176 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:27.300349 21990 hybrid_clock.cc:648] HybridClock initialized: now 1790693187300325 us; error 40 us; skew 500 ppm
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[21977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:27.310547 21990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:27.311582 21990 webserver.cc:460] Webserver started at http://127.0.98.62:43167/ using document root <none> and password file <none>
I20260929 14:46:27.312211 21990 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:27.312290 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:27.312512 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:27.314754 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "4a9a66fbe1154b81acf35168408f4612"
format_stamp: "Formatted at 2026-09-29 14:46:27 on dist-test-slave-g8hp"
server_key: "e0ca403c3268b7b2f68adb5df7a9f4b8"
server_key_iv: "44f79eb7443ffd3b012d0bf0ee408e0f"
server_key_version: "encryptionkey@0"
I20260929 14:46:27.315335 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a9a66fbe1154b81acf35168408f4612"
format_stamp: "Formatted at 2026-09-29 14:46:27 on dist-test-slave-g8hp"
server_key: "e0ca403c3268b7b2f68adb5df7a9f4b8"
server_key_iv: "44f79eb7443ffd3b012d0bf0ee408e0f"
server_key_version: "encryptionkey@0"
I20260929 14:46:27.319193 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:27.321851 22005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:27.323153 21990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:46:27.323275 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "4a9a66fbe1154b81acf35168408f4612"
format_stamp: "Formatted at 2026-09-29 14:46:27 on dist-test-slave-g8hp"
server_key: "e0ca403c3268b7b2f68adb5df7a9f4b8"
server_key_iv: "44f79eb7443ffd3b012d0bf0ee408e0f"
server_key_version: "encryptionkey@0"
I20260929 14:46:27.323391 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:27.338742 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:27.342147 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:27.342332 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:27.349304 21990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40797
I20260929 14:46:27.349354 22057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40797 every 8 connection(s)
I20260929 14:46:27.350257 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:27.350955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 21990
I20260929 14:46:27.351074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:27.351321 392 external_mini_cluster.cc:1428] Setting key cae06a1618429d98dca0f177dd83de92
I20260929 14:46:27.353515 22058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:27.359153 22058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612: Bootstrap starting.
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[21977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:27.361508 22058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:27.362217 22058 log.cc:824] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:27.364066 22058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612: No bootstrap required, opened a new log
I20260929 14:46:27.364970 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:27.352653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53091 (local address 127.0.98.62:40797)
0929 14:46:27.353126 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:27.353140 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:27.353175 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:46:27.353878 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:27.353898 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:27.354246 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:27.354579 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:27.355396 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:27.356431 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:27.357330 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:27.357801 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:27.357944 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:27.360460 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:27.360465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:27.360478 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:27.360507 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:27.362085 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:27.362609 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:27.362614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:27.362618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:27.362728 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:27.363044 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:27.363047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:27.363048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:27.363449 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:27.363642 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:27.363956 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 14:46:27.364187 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":167,"threads_started":1}
I20260929 14:46:27.367748 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:27.367724 22058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260929 14:46:27.367969 22058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:27.368026 22058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9a66fbe1154b81acf35168408f4612, State: Initialized, Role: FOLLOWER
I20260929 14:46:27.368423 22058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260929 14:46:27.368541 22058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:27.368597 22058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:27.368685 22058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:27.369652 22058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260929 14:46:27.370018 22058 leader_election.cc:302] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a9a66fbe1154b81acf35168408f4612; no voters:
I20260929 14:46:27.370357 22058 leader_election.cc:288] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:27.370493 22068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:27.370714 22068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [term 1 LEADER]: Becoming Leader. State: Replica: 4a9a66fbe1154b81acf35168408f4612, State: Running, Role: LEADER
I20260929 14:46:27.371097 22068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } }
I20260929 14:46:27.371716 22058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:27.372246 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9a66fbe1154b81acf35168408f4612. Latest consensus state: current_term: 1 leader_uuid: "4a9a66fbe1154b81acf35168408f4612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260929 14:46:27.372430 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:27.372918 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:27.373359 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a9a66fbe1154b81acf35168408f4612" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a66fbe1154b81acf35168408f4612" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40797 } } }
I20260929 14:46:27.373456 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:27.376302 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:27.379315 22061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:27.369825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33979 (local address 127.0.98.62:40797)
0929 14:46:27.369930 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:27.369934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:27.370056 (+ 122us) server_negotiation.cc:407] Connection header received
0929 14:46:27.370161 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:27.370164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:27.370211 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:27.370291 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:27.371073 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:27.371844 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:27.373590 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:27.373914 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:27.373996 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:27.375873 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:27.375877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:27.375880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:27.375907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:27.377549 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:27.378018 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:27.378022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:27.378024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:27.378098 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:27.378394 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:27.378398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:27.378400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:27.378599 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:27.378737 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:27.379060 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 14:46:27.379156 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:46:27.382709 22076 catalog_manager.cc:1347] Generated new cluster ID: c61e28f9087446159e0b76621c1556f8
I20260929 14:46:27.382781 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:27.403566 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:27.404597 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:27.415835 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a9a66fbe1154b81acf35168408f4612: Generated new TSK 0
I20260929 14:46:27.416615 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:27.425961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 21990
W20260929 14:46:27.430572 21975 connection.cc:535] client connection to 127.0.98.62:40797 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40797): Connection reset by peer (error 104)
2026-09-29T14:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[22089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[22089](info): set up 2 sockets
Sep 29 14:46:27 dist-test-slave-g8hp krb5kdc[22089](info): commencing operation
krb5kdc: starting...
W20260929 14:46:28.385053 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:28 dist-test-slave-g8hp krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:28.509152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:28.621299 22102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:28.621589 22102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:28.621651 22102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:28.625766 22102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:28.625841 22102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:28.625867 22102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:28.625890 22102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:28.625918 22102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:28.631392 22102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:28.632450 22102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:28.633507 22102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:28.642153 22110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:28.642318 22102 server_base.cc:1034] running on GCE node
W20260929 14:46:28.642124 22108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:28.642256 22107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:28.643081 22102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:28.644330 22102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:28.645519 22102 hybrid_clock.cc:648] HybridClock initialized: now 1790693188645444 us; error 89 us; skew 500 ppm
Sep 29 14:46:28 dist-test-slave-g8hp krb5kdc[22089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:28.648373 22102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:28.649385 22102 webserver.cc:460] Webserver started at http://127.0.98.62:35669/ using document root <none> and password file <none>
I20260929 14:46:28.649930 22102 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:28.649996 22102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:28.650185 22102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:28.652287 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "aa0f2bc807664966b2c0dae5af452cae"
format_stamp: "Formatted at 2026-09-29 14:46:28 on dist-test-slave-g8hp"
server_key: "76ca9984031a8a0475a1b39282604a9d"
server_key_iv: "4905c86009664fa97713c9a9ece723fe"
server_key_version: "encryptionkey@0"
I20260929 14:46:28.652829 22102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa0f2bc807664966b2c0dae5af452cae"
format_stamp: "Formatted at 2026-09-29 14:46:28 on dist-test-slave-g8hp"
server_key: "76ca9984031a8a0475a1b39282604a9d"
server_key_iv: "4905c86009664fa97713c9a9ece723fe"
server_key_version: "encryptionkey@0"
I20260929 14:46:28.656711 22102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:28.659318 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:28.660593 22102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:46:28.660820 22102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "aa0f2bc807664966b2c0dae5af452cae"
format_stamp: "Formatted at 2026-09-29 14:46:28 on dist-test-slave-g8hp"
server_key: "76ca9984031a8a0475a1b39282604a9d"
server_key_iv: "4905c86009664fa97713c9a9ece723fe"
server_key_version: "encryptionkey@0"
I20260929 14:46:28.660991 22102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:28.678167 22102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:28.681421 22102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:28.681612 22102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:28.688647 22102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33863
I20260929 14:46:28.688696 22169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33863 every 8 connection(s)
I20260929 14:46:28.689657 22102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:28.692523 22170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:28.695887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22102
I20260929 14:46:28.696026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:28.696280 392 external_mini_cluster.cc:1428] Setting key 5ce0b3ae2930a02e5f8b99b8a84a60b7
I20260929 14:46:28.698552 22170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae: Bootstrap starting.
I20260929 14:46:28.700925 22170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:28.701658 22170 log.cc:824] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:28.703878 22170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae: No bootstrap required, opened a new log
I20260929 14:46:28.705595 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:28.697768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57249 (local address 127.0.98.62:33863)
0929 14:46:28.698210 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:28.698226 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:28.698261 (+ 35us) server_negotiation.cc:407] Connection header received
0929 14:46:28.698953 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:28.698972 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:28.699338 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:28.699623 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:28.699983 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:28.701020 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:28.701916 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:28.702479 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:28.702631 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:28.703526 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:28.703533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:28.703549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:28.704068 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:28.704137 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:28.704178 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:28.704315 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:28.704493 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 14:46:28.704629 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":130,"threads_started":1}
I20260929 14:46:28.708163 22170 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } }
I20260929 14:46:28.708454 22170 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:28.708516 22170 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0f2bc807664966b2c0dae5af452cae, State: Initialized, Role: FOLLOWER
I20260929 14:46:28.708662 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:28.709064 22170 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } }
I20260929 14:46:28.709290 22170 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:28.709359 22170 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:28.709463 22170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:28.710660 22170 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } }
I20260929 14:46:28.711007 22170 leader_election.cc:302] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0f2bc807664966b2c0dae5af452cae; no voters:
I20260929 14:46:28.711333 22170 leader_election.cc:288] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:28.711417 22180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:28.711618 22180 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [term 1 LEADER]: Becoming Leader. State: Replica: aa0f2bc807664966b2c0dae5af452cae, State: Running, Role: LEADER
I20260929 14:46:28.711992 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } }
I20260929 14:46:28.712684 22170 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:28.713639 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0f2bc807664966b2c0dae5af452cae. Latest consensus state: current_term: 1 leader_uuid: "aa0f2bc807664966b2c0dae5af452cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } } }
I20260929 14:46:28.713775 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:28.714143 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:28.714093 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0f2bc807664966b2c0dae5af452cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0f2bc807664966b2c0dae5af452cae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33863 } } }
I20260929 14:46:28.714241 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:28.715842 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:28.711023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48635 (local address 127.0.98.62:33863)
0929 14:46:28.711148 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:28.711152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:28.711360 (+ 208us) server_negotiation.cc:407] Connection header received
0929 14:46:28.711489 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:28.711493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:28.711542 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:28.711604 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:28.711966 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:28.712548 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:28.713647 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:28.713985 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:28.714088 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:28.715174 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:28.715178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:28.715180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:28.715193 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:28.715221 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:28.715232 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:28.715359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:28.715661 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 14:46:28.715698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 14:46:28.717726 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:28.725524 22189 catalog_manager.cc:1347] Generated new cluster ID: 7f5cca3deff245deb3acfcb81f811eb4
I20260929 14:46:28.725591 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:28.734460 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:28.735512 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:28.740588 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0f2bc807664966b2c0dae5af452cae: Generated new TSK 0
I20260929 14:46:28.741297 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:28.746716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22102
W20260929 14:46:28.751623 22087 connection.cc:535] client connection to 127.0.98.62:33863 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33863): Connection reset by peer (error 104)
2026-09-29T14:46:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:28 dist-test-slave-g8hp krb5kdc[22201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:28 dist-test-slave-g8hp krb5kdc[22201](info): set up 2 sockets
Sep 29 14:46:28 dist-test-slave-g8hp krb5kdc[22201](info): commencing operation
krb5kdc: starting...
W20260929 14:46:29.712839 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:29 dist-test-slave-g8hp krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:29.836540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:29.947160 22214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:29.947496 22214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:29.947563 22214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:29.951995 22214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:29.952066 22214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:29.952091 22214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:29.952112 22214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:29.952142 22214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:29.957681 22214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:29.958762 22214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:29.959774 22214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:29.968422 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:29.968462 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:29.968432 22222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:29.968807 22214 server_base.cc:1034] running on GCE node
I20260929 14:46:29.969365 22214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:29.970505 22214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:29.971678 22214 hybrid_clock.cc:648] HybridClock initialized: now 1790693189971654 us; error 35 us; skew 500 ppm
Sep 29 14:46:29 dist-test-slave-g8hp krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:29.974812 22214 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:29.975736 22214 webserver.cc:460] Webserver started at http://127.0.98.62:45615/ using document root <none> and password file <none>
I20260929 14:46:29.976279 22214 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:29.976349 22214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:29.976547 22214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:29.978611 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "22e81d77ee204fddaef9e538c8a6e4ec"
format_stamp: "Formatted at 2026-09-29 14:46:29 on dist-test-slave-g8hp"
server_key: "cec9be1687cd6443d6df49605502b9d2"
server_key_iv: "a4732eadecbc2de36fc0541560a3a98a"
server_key_version: "encryptionkey@0"
I20260929 14:46:29.979156 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "22e81d77ee204fddaef9e538c8a6e4ec"
format_stamp: "Formatted at 2026-09-29 14:46:29 on dist-test-slave-g8hp"
server_key: "cec9be1687cd6443d6df49605502b9d2"
server_key_iv: "a4732eadecbc2de36fc0541560a3a98a"
server_key_version: "encryptionkey@0"
I20260929 14:46:29.983047 22214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:29.985724 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:29.987155 22214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:29.987290 22214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "22e81d77ee204fddaef9e538c8a6e4ec"
format_stamp: "Formatted at 2026-09-29 14:46:29 on dist-test-slave-g8hp"
server_key: "cec9be1687cd6443d6df49605502b9d2"
server_key_iv: "a4732eadecbc2de36fc0541560a3a98a"
server_key_version: "encryptionkey@0"
I20260929 14:46:29.987406 22214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:29.992177 22214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:29.995517 22214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:29.995711 22214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:30.002534 22214 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42517
I20260929 14:46:30.002579 22281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42517 every 8 connection(s)
I20260929 14:46:30.003526 22214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:30.006780 22282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:30.012591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22214
I20260929 14:46:30.012737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:30.012782 22282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec: Bootstrap starting.
I20260929 14:46:30.013032 392 external_mini_cluster.cc:1428] Setting key e4e3943cade74e69fcf5634a7f2893f8
I20260929 14:46:30.015596 22282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:30.016530 22282 log.cc:824] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:30.018887 22282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec: No bootstrap required, opened a new log
I20260929 14:46:30.018981 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:30.014484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39239 (local address 127.0.98.62:42517)
0929 14:46:30.014971 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:30.014990 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:30.015034 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:46:30.015929 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:30.015955 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:30.016359 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:30.016750 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:30.017563 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:30.017569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:30.017585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:30.017597 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:30.017662 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:30.017689 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:30.017791 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:30.017912 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 14:46:30.018067 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":142,"threads_started":1}
I20260929 14:46:30.022280 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:30.023357 22282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260929 14:46:30.023748 22282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:30.023833 22282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22e81d77ee204fddaef9e538c8a6e4ec, State: Initialized, Role: FOLLOWER
I20260929 14:46:30.024369 22282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260929 14:46:30.024528 22282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:30.024641 22282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:30.024752 22282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:30.025929 22282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260929 14:46:30.026307 22282 leader_election.cc:302] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22e81d77ee204fddaef9e538c8a6e4ec; no voters:
I20260929 14:46:30.026557 22285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:30.024682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45585 (local address 127.0.98.62:42517)
0929 14:46:30.024853 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:30.024885 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:30.025045 (+ 160us) server_negotiation.cc:407] Connection header received
0929 14:46:30.025171 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:30.025174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:30.025215 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:30.025291 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:30.026154 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:30.026158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:30.026160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:30.026171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:30.026199 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:30.026211 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:30.026286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:30.026379 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 14:46:30.026421 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:46:30.026659 22282 leader_election.cc:288] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:30.026705 22292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:30.026927 22292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [term 1 LEADER]: Becoming Leader. State: Replica: 22e81d77ee204fddaef9e538c8a6e4ec, State: Running, Role: LEADER
I20260929 14:46:30.027307 22292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } }
I20260929 14:46:30.028095 22282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:30.029173 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22e81d77ee204fddaef9e538c8a6e4ec. Latest consensus state: current_term: 1 leader_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } } }
I20260929 14:46:30.029202 22294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22e81d77ee204fddaef9e538c8a6e4ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42517 } } }
I20260929 14:46:30.029418 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:30.029425 22294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:30.030514 22301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:30.033241 22301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:30.038727 22301 catalog_manager.cc:1347] Generated new cluster ID: b6f0229748994e8282ef3800a75d7708
I20260929 14:46:30.038801 22301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:30.054502 22301 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:30.054952 22301 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:30.060741 22301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22e81d77ee204fddaef9e538c8a6e4ec: Generated new TSK 0
I20260929 14:46:30.061517 22301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:30.066182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22214
2026-09-29T14:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:30 dist-test-slave-g8hp krb5kdc[22313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:30 dist-test-slave-g8hp krb5kdc[22313](info): set up 2 sockets
Sep 29 14:46:30 dist-test-slave-g8hp krb5kdc[22313](info): commencing operation
krb5kdc: starting...
W20260929 14:46:31.033013 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:31.161379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:31.271459 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:31.271754 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:31.271808 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:31.275879 22326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:31.275959 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:31.275977 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:31.275992 22326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:31.276007 22326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:31.281551 22326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:31.282604 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:31.283576 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:31.292517 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:31.292495 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:31.292495 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:31.292717 22326 server_base.cc:1034] running on GCE node
I20260929 14:46:31.293336 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:31.294543 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:31.295739 22326 hybrid_clock.cc:648] HybridClock initialized: now 1790693191295707 us; error 47 us; skew 500 ppm
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:31.298666 22326 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:31.299803 22326 webserver.cc:460] Webserver started at http://127.0.98.62:44417/ using document root <none> and password file <none>
I20260929 14:46:31.300396 22326 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:31.300475 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:31.300685 22326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:31.302942 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "019e82dfce9543e8a75306961bba8cca"
format_stamp: "Formatted at 2026-09-29 14:46:31 on dist-test-slave-g8hp"
server_key: "cb533621eced43f47c0515d8d40343ed"
server_key_iv: "45fdd1b1e35648b2de106429edf49486"
server_key_version: "encryptionkey@0"
I20260929 14:46:31.303505 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "019e82dfce9543e8a75306961bba8cca"
format_stamp: "Formatted at 2026-09-29 14:46:31 on dist-test-slave-g8hp"
server_key: "cb533621eced43f47c0515d8d40343ed"
server_key_iv: "45fdd1b1e35648b2de106429edf49486"
server_key_version: "encryptionkey@0"
I20260929 14:46:31.307498 22326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:31.310317 22341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:31.311794 22326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:46:31.311934 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "019e82dfce9543e8a75306961bba8cca"
format_stamp: "Formatted at 2026-09-29 14:46:31 on dist-test-slave-g8hp"
server_key: "cb533621eced43f47c0515d8d40343ed"
server_key_iv: "45fdd1b1e35648b2de106429edf49486"
server_key_version: "encryptionkey@0"
I20260929 14:46:31.312049 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:31.338634 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:31.342157 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:31.342345 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:31.349563 22326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34449
I20260929 14:46:31.349583 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34449 every 8 connection(s)
I20260929 14:46:31.350610 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:31.353494 22394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:31.358088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22326
I20260929 14:46:31.358237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:31.358492 392 external_mini_cluster.cc:1428] Setting key e1791c0bc6c769de562f3ff2fe2969c7
I20260929 14:46:31.359752 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca: Bootstrap starting.
I20260929 14:46:31.362239 22394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:31.363183 22394 log.cc:824] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:31.365485 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca: No bootstrap required, opened a new log
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:31.369958 22394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260929 14:46:31.370249 22394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:31.370314 22394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019e82dfce9543e8a75306961bba8cca, State: Initialized, Role: FOLLOWER
I20260929 14:46:31.370782 22394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260929 14:46:31.370905 22394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:31.370975 22394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:31.371073 22394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:31.372282 22394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260929 14:46:31.372673 22394 leader_election.cc:302] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 019e82dfce9543e8a75306961bba8cca; no voters:
I20260929 14:46:31.373061 22394 leader_election.cc:288] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:31.373152 22399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:31.373325 22399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [term 1 LEADER]: Becoming Leader. State: Replica: 019e82dfce9543e8a75306961bba8cca, State: Running, Role: LEADER
I20260929 14:46:31.373708 22399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260929 14:46:31.374104 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:31.360094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59867 (local address 127.0.98.62:34449)
0929 14:46:31.360557 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:31.360576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:31.360612 (+ 36us) server_negotiation.cc:407] Connection header received
0929 14:46:31.361402 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:31.361421 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:31.361793 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:31.362077 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:31.363043 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:31.364295 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:31.365482 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:31.365985 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:31.366106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:31.368990 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:31.368995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:31.369009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:31.369038 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:31.370972 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:31.371471 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:31.371475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:31.371480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:31.371592 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:31.371854 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:31.371857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:31.371858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:31.372285 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:31.372453 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:31.372765 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 14:46:31.373171 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":131,"threads_started":1}
I20260929 14:46:31.374428 22394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:31.375728 22401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 019e82dfce9543e8a75306961bba8cca. Latest consensus state: current_term: 1 leader_uuid: "019e82dfce9543e8a75306961bba8cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } } }
I20260929 14:46:31.375916 22401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:31.376673 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "019e82dfce9543e8a75306961bba8cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "019e82dfce9543e8a75306961bba8cca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } } }
I20260929 14:46:31.376766 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:31.376806 22414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:31.377543 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:31.379539 22414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:31.385140 22414 catalog_manager.cc:1347] Generated new cluster ID: c55787417c304a82b8dca5a8d35e8b05
I20260929 14:46:31.385217 22414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:31.389086 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:31.379621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58647 (local address 127.0.98.62:34449)
0929 14:46:31.379726 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:31.379730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:31.380014 (+ 284us) server_negotiation.cc:407] Connection header received
0929 14:46:31.380273 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:31.380276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:31.380328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:31.380414 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:31.381213 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:31.382008 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:31.383188 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:31.383563 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:31.383639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:31.385508 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:31.385513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:31.385515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:31.385543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:31.387164 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:31.387657 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:31.387661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:31.387663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:31.387761 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:31.388062 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:31.388066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:31.388068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:31.388301 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:31.388402 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:31.388758 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 14:46:31.388913 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:46:31.395406 22414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:31.396410 22414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:31.403234 22414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 019e82dfce9543e8a75306961bba8cca: Generated new TSK 0
I20260929 14:46:31.404075 22414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:31.411753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22326
2026-09-29T14:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22425](info): set up 2 sockets
Sep 29 14:46:31 dist-test-slave-g8hp krb5kdc[22425](info): commencing operation
krb5kdc: starting...
W20260929 14:46:32.380923 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:32 dist-test-slave-g8hp krb5kdc[22425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:32.505750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:32.614610 22438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:32.614904 22438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:32.614964 22438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:32.618994 22438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:32.619076 22438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:32.619103 22438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:32.619125 22438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:32.619146 22438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:32.624516 22438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:32.625628 22438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:32.626648 22438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:32.635056 22443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:32.635111 22438 server_base.cc:1034] running on GCE node
W20260929 14:46:32.635133 22444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:32.635087 22446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:32.635872 22438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:32.636986 22438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:32.638157 22438 hybrid_clock.cc:648] HybridClock initialized: now 1790693192638132 us; error 39 us; skew 500 ppm
Sep 29 14:46:32 dist-test-slave-g8hp krb5kdc[22425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:32.646405 22438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:32.647425 22438 webserver.cc:460] Webserver started at http://127.0.98.62:38303/ using document root <none> and password file <none>
I20260929 14:46:32.647987 22438 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:32.648056 22438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:32.648258 22438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:32.650465 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "111f92def0d34c4ead186bc689c1dc2d"
format_stamp: "Formatted at 2026-09-29 14:46:32 on dist-test-slave-g8hp"
server_key: "6f15916bba4541236caf102e9fe3b11e"
server_key_iv: "151cdbd1a5e29c28b5953a2133bde4e9"
server_key_version: "encryptionkey@0"
I20260929 14:46:32.651062 22438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "111f92def0d34c4ead186bc689c1dc2d"
format_stamp: "Formatted at 2026-09-29 14:46:32 on dist-test-slave-g8hp"
server_key: "6f15916bba4541236caf102e9fe3b11e"
server_key_iv: "151cdbd1a5e29c28b5953a2133bde4e9"
server_key_version: "encryptionkey@0"
I20260929 14:46:32.655007 22438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 14:46:32.657618 22453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:32.658859 22438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:46:32.658979 22438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "111f92def0d34c4ead186bc689c1dc2d"
format_stamp: "Formatted at 2026-09-29 14:46:32 on dist-test-slave-g8hp"
server_key: "6f15916bba4541236caf102e9fe3b11e"
server_key_iv: "151cdbd1a5e29c28b5953a2133bde4e9"
server_key_version: "encryptionkey@0"
I20260929 14:46:32.659086 22438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:32.677064 22438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:32.680249 22438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:32.680438 22438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:32.687629 22438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35253
I20260929 14:46:32.687652 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35253 every 8 connection(s)
I20260929 14:46:32.688613 22438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:32.691641 22506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:32.692288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22438
I20260929 14:46:32.692435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:32.692739 392 external_mini_cluster.cc:1428] Setting key 453fbb41906f6b0946853a04b5c99b34
I20260929 14:46:32.698084 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d: Bootstrap starting.
I20260929 14:46:32.700693 22506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:32.701524 22506 log.cc:824] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:32.702054 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:32.694143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49639 (local address 127.0.98.62:35253)
0929 14:46:32.694561 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:32.694576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:32.694613 (+ 37us) server_negotiation.cc:407] Connection header received
0929 14:46:32.695282 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:32.695300 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:32.695657 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:32.695962 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:32.696361 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:32.697406 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:32.698566 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:32.699093 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:32.699231 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:32.700116 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:32.700121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:32.700136 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:32.700579 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:32.700639 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:32.700664 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:32.700750 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:32.700981 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 14:46:32.701215 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260929 14:46:32.703570 22506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d: No bootstrap required, opened a new log
I20260929 14:46:32.705225 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:32.708065 22506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } }
I20260929 14:46:32.708376 22506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:32.708443 22506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111f92def0d34c4ead186bc689c1dc2d, State: Initialized, Role: FOLLOWER
I20260929 14:46:32.708950 22506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } }
I20260929 14:46:32.709095 22506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:32.709159 22506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:32.709266 22506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:32.710249 22506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } }
I20260929 14:46:32.710613 22506 leader_election.cc:302] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 111f92def0d34c4ead186bc689c1dc2d; no voters:
I20260929 14:46:32.710943 22506 leader_election.cc:288] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:32.711069 22516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:32.711330 22516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [term 1 LEADER]: Becoming Leader. State: Replica: 111f92def0d34c4ead186bc689c1dc2d, State: Running, Role: LEADER
I20260929 14:46:32.711776 22516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } }
I20260929 14:46:32.712054 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:32.707737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51569 (local address 127.0.98.62:35253)
0929 14:46:32.707857 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:32.707862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:32.707878 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:46:32.708100 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:32.708103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:32.708148 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:32.708236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:32.708537 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:32.709315 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:32.710121 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:32.710401 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:32.710495 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:32.711396 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:32.711400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:32.711403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:32.711418 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:32.711448 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:32.711460 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:32.711528 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:32.711866 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 14:46:32.711924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:46:32.712466 22506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:32.713716 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "111f92def0d34c4ead186bc689c1dc2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } } }
I20260929 14:46:32.713829 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:32.714144 22518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 111f92def0d34c4ead186bc689c1dc2d. Latest consensus state: current_term: 1 leader_uuid: "111f92def0d34c4ead186bc689c1dc2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "111f92def0d34c4ead186bc689c1dc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35253 } } }
I20260929 14:46:32.714224 22518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:32.714211 22525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:32.717152 22525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:32.723443 22525 catalog_manager.cc:1347] Generated new cluster ID: 25275a96f205474c9bd8baebe52b0e93
I20260929 14:46:32.723531 22525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:32.735651 22525 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:32.737537 22525 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:32.745304 22525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 111f92def0d34c4ead186bc689c1dc2d: Generated new TSK 0
I20260929 14:46:32.746078 22525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:32.750974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22438
2026-09-29T14:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:32 dist-test-slave-g8hp krb5kdc[22537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:32 dist-test-slave-g8hp krb5kdc[22537](info): set up 2 sockets
Sep 29 14:46:32 dist-test-slave-g8hp krb5kdc[22537](info): commencing operation
krb5kdc: starting...
W20260929 14:46:33.713016 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:33 dist-test-slave-g8hp krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:33.835510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:33.946249 22550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:33.946550 22550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:33.946593 22550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:33.950699 22550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:33.950780 22550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:33.950798 22550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:33.950814 22550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:33.950829 22550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:33.956511 22550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:33.957684 22550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:33.958765 22550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:33.967886 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:33.967904 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:33.967911 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:33.967971 22550 server_base.cc:1034] running on GCE node
I20260929 14:46:33.968731 22550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:33.969871 22550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:33.971068 22550 hybrid_clock.cc:648] HybridClock initialized: now 1790693193971040 us; error 42 us; skew 500 ppm
Sep 29 14:46:33 dist-test-slave-g8hp krb5kdc[22537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:33.973958 22550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:33.974970 22550 webserver.cc:460] Webserver started at http://127.0.98.62:46801/ using document root <none> and password file <none>
I20260929 14:46:33.975517 22550 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:33.975586 22550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:33.975780 22550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:33.977968 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "f6affa98c25b4e05b350993888e01ea1"
format_stamp: "Formatted at 2026-09-29 14:46:33 on dist-test-slave-g8hp"
server_key: "442ac0145deaf16c2fd4eec7a1b997ca"
server_key_iv: "81a73293c97c4818b00d413a3a03f3ef"
server_key_version: "encryptionkey@0"
I20260929 14:46:33.978546 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6affa98c25b4e05b350993888e01ea1"
format_stamp: "Formatted at 2026-09-29 14:46:33 on dist-test-slave-g8hp"
server_key: "442ac0145deaf16c2fd4eec7a1b997ca"
server_key_iv: "81a73293c97c4818b00d413a3a03f3ef"
server_key_version: "encryptionkey@0"
I20260929 14:46:33.982378 22550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:33.985085 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:33.986433 22550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:33.986567 22550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "f6affa98c25b4e05b350993888e01ea1"
format_stamp: "Formatted at 2026-09-29 14:46:33 on dist-test-slave-g8hp"
server_key: "442ac0145deaf16c2fd4eec7a1b997ca"
server_key_iv: "81a73293c97c4818b00d413a3a03f3ef"
server_key_version: "encryptionkey@0"
I20260929 14:46:33.986696 22550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:33.991662 22550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:33.995097 22550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:33.995283 22550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:34.002125 22550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35419
I20260929 14:46:34.002149 22617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35419 every 8 connection(s)
I20260929 14:46:34.003070 22550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:34.006300 22618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:34.012012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22550
I20260929 14:46:34.012172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:34.012329 22618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1: Bootstrap starting.
I20260929 14:46:34.012454 392 external_mini_cluster.cc:1428] Setting key 6e00ea3e77c0db4605fec4ed8b93bde0
I20260929 14:46:34.015122 22618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:34.015985 22618 log.cc:824] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:34.017917 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:34.013880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52913 (local address 127.0.98.62:35419)
0929 14:46:34.014340 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:34.014355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:34.014393 (+ 38us) server_negotiation.cc:407] Connection header received
0929 14:46:34.015070 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:34.015089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:34.015442 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:34.015754 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:34.016607 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:34.016614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:34.016632 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:34.016645 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:34.016722 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:34.016752 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:34.016889 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:34.017010 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 14:46:34.017181 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":135,"threads_started":1}
I20260929 14:46:34.018293 22618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1: No bootstrap required, opened a new log
I20260929 14:46:34.020749 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:34.022703 22618 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260929 14:46:34.023070 22618 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:46:34.023144 22618 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6affa98c25b4e05b350993888e01ea1, State: Initialized, Role: FOLLOWER
I20260929 14:46:34.023779 22618 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260929 14:46:34.023959 22618 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:46:34.024052 22618 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:46:34.024163 22618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:34.025277 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:34.023418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49135 (local address 127.0.98.62:35419)
0929 14:46:34.023544 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:34.023549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:34.023803 (+ 254us) server_negotiation.cc:407] Connection header received
0929 14:46:34.023887 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:34.023890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:34.023930 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:34.024024 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:34.024895 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:34.024899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:34.024902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 14:46:34.024914 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:34.024945 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 14:46:34.024958 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:34.025044 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:34.025114 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 14:46:34.025156 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 14:46:34.025419 22618 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260929 14:46:34.025804 22618 leader_election.cc:302] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6affa98c25b4e05b350993888e01ea1; no voters:
I20260929 14:46:34.026160 22618 leader_election.cc:288] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:46:34.026250 22628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:34.026494 22628 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [term 1 LEADER]: Becoming Leader. State: Replica: f6affa98c25b4e05b350993888e01ea1, State: Running, Role: LEADER
I20260929 14:46:34.026861 22628 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260929 14:46:34.027493 22618 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:34.027992 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6affa98c25b4e05b350993888e01ea1. Latest consensus state: current_term: 1 leader_uuid: "f6affa98c25b4e05b350993888e01ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260929 14:46:34.028108 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:34.028375 22629 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6affa98c25b4e05b350993888e01ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6affa98c25b4e05b350993888e01ea1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260929 14:46:34.028473 22629 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:34.029110 22632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:34.032014 22632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:34.037706 22632 catalog_manager.cc:1347] Generated new cluster ID: 187b6918493846ccbc4ae89652c54a36
I20260929 14:46:34.037807 22632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:34.047209 22632 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:34.047518 22632 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:34.057271 22632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6affa98c25b4e05b350993888e01ea1: Generated new TSK 0
I20260929 14:46:34.058156 22632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:46:34.068822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22550
2026-09-29T14:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1317 ms)
[----------] 18 tests from AuthTokenIssuingTest (24189 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:34 dist-test-slave-g8hp krb5kdc[22649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:34 dist-test-slave-g8hp krb5kdc[22649](info): set up 2 sockets
Sep 29 14:46:34 dist-test-slave-g8hp krb5kdc[22649](info): commencing operation
krb5kdc: starting...
W20260929 14:46:35.037122 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:35.164227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:35.275692 22662 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:35.276021 22662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:35.276058 22662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:35.276122 22662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:35.280267 22662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:35.280337 22662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:35.280364 22662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:35.280386 22662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:35.280414 22662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:35.285933 22662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:35.287060 22662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:35.288084 22662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:35.297070 22667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:35.297080 22662 server_base.cc:1034] running on GCE node
W20260929 14:46:35.297031 22670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:35.297031 22668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:35.297827 22662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:35.298925 22662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:35.300094 22662 hybrid_clock.cc:648] HybridClock initialized: now 1790693195300062 us; error 40 us; skew 500 ppm
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:35.303153 22662 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:35.304127 22662 webserver.cc:460] Webserver started at http://127.0.98.62:42309/ using document root <none> and password file <none>
I20260929 14:46:35.304790 22662 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:35.304888 22662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:35.305107 22662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:35.307206 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "737c22a3cae24712830bb246ca3ae7b3"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "c1fa844085c1478104b2847aaf134703"
server_key_iv: "08a756430771ae82dea021702ca98d82"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.307757 22662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "737c22a3cae24712830bb246ca3ae7b3"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "c1fa844085c1478104b2847aaf134703"
server_key_iv: "08a756430771ae82dea021702ca98d82"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.311717 22662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 14:46:35.314314 22677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:35.315680 22662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 14:46:35.315862 22662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "737c22a3cae24712830bb246ca3ae7b3"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "c1fa844085c1478104b2847aaf134703"
server_key_iv: "08a756430771ae82dea021702ca98d82"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.315970 22662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:35.365931 22662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:35.369117 22662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:35.369307 22662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:35.376101 22662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36199
I20260929 14:46:35.376350 22729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36199 every 8 connection(s)
I20260929 14:46:35.377103 22662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:35.379907 22730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:35.381417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22662
I20260929 14:46:35.381537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:35.381834 392 external_mini_cluster.cc:1428] Setting key ebd0ae6aafeb6dab2e98ae5085396d29
I20260929 14:46:35.384441 22730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:35.400370 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.386762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36199 (local address 127.0.0.1:59658)
0929 14:46:35.387246 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.387297 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.388222 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.389087 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.389092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.389514 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.389799 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.389813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.391044 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.391047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.391971 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.391981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.392643 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.392648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.392759 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.393324 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.393351 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.395102 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.397320 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.397329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.397338 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.397564 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.397948 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.397951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.397952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.398066 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.398831 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.398837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.398969 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.399113 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.399568 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":131,"thread_start_us":104,"threads_started":1}
I20260929 14:46:35.400372 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.387650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.0.98.62:36199)
0929 14:46:35.387986 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.387997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.388055 (+ 58us) server_negotiation.cc:407] Connection header received
0929 14:46:35.388550 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.388556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.388820 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.389080 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.389953 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.390918 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.392090 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.392472 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.392635 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.395253 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.395258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.395269 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.395290 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.397207 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.397682 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.397689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.397691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.397842 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.398154 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.398157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.398159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.398707 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.398846 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.399213 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.399515 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260929 14:46:35.403071 22730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:35.403932 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.403594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52592)
0929 14:46:35.403758 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.403858 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 14:46:35.404175 22679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
I20260929 14:46:35.405309 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.405037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52606)
0929 14:46:35.405181 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.405248 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 14:46:35.405435 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:46:35.413239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33727
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:46:35.454828 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } attempt: 1
I20260929 14:46:35.455801 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.455457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52616)
0929 14:46:35.455624 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.455706 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:46:35.456574 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.456358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52622)
0929 14:46:35.456456 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.456508 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 14:46:35.456702 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
I20260929 14:46:35.524895 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } attempt: 2
I20260929 14:46:35.525911 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.525554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52630)
0929 14:46:35.525729 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.525799 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 14:46:35.526897 22737 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:35.527324 22737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:35.527376 22737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:35.527442 22737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:35.533273 22737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:35.533399 22737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:35.533444 22737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:35.533480 22737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:35.533515 22737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:35.535704 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.535286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52642)
0929 14:46:35.535466 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.535558 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 14:46:35.535898 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33727: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33727: connect: Connection refused (error 111)
I20260929 14:46:35.539901 22737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:35.541065 22737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:35.542140 22737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:35.550972 22745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:35.550974 22742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:35.550974 22743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:35.551196 22737 server_base.cc:1034] running on GCE node
I20260929 14:46:35.551798 22737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:35.552918 22737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:35.554091 22737 hybrid_clock.cc:648] HybridClock initialized: now 1790693195554067 us; error 37 us; skew 500 ppm
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:35.557075 22737 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:46:35.557994 22737 webserver.cc:460] Webserver started at http://127.0.98.61:43193/ using document root <none> and password file <none>
I20260929 14:46:35.558531 22737 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:35.558596 22737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:35.558800 22737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:35.560930 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data/instance:
uuid: "3f497c1ecec742a1b32166666f32faa7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "b6f5e7d1a22194bcb27e4bdf0a936ee7"
server_key_iv: "c41c47c6645657bf4bbece6f3e43e201"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.561498 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal/instance:
uuid: "3f497c1ecec742a1b32166666f32faa7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "b6f5e7d1a22194bcb27e4bdf0a936ee7"
server_key_iv: "c41c47c6645657bf4bbece6f3e43e201"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.565608 22737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:35.568351 22752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:35.569739 22737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:35.569862 22737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "3f497c1ecec742a1b32166666f32faa7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "b6f5e7d1a22194bcb27e4bdf0a936ee7"
server_key_iv: "c41c47c6645657bf4bbece6f3e43e201"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.569965 22737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:35.582214 22737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:35.585479 22737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:35.585672 22737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:35.593067 22737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33727
I20260929 14:46:35.593394 22804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33727 every 8 connection(s)
I20260929 14:46:35.594058 22737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:46:35.597651 22805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:35.600252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22737
I20260929 14:46:35.600384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-1/wal/instance
I20260929 14:46:35.600653 392 external_mini_cluster.cc:1428] Setting key 9cdfcdfb880bbe96985461f520b944cd
I20260929 14:46:35.602862 22805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:35.618846 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.605313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.0.98.62:36199)
0929 14:46:35.606962 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.606968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.606988 (+ 20us) server_negotiation.cc:407] Connection header received
0929 14:46:35.607919 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.607923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.607986 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.608099 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.609832 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.610383 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.612188 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.612526 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.612674 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.615267 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.615270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.615272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.615299 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.616694 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.617267 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.617270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.617272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.617350 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.617696 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.617699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.617700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.617891 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.617977 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.618540 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.618672 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1541}
I20260929 14:46:35.619827 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.605876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36199 (local address 127.0.0.1:59660)
0929 14:46:35.606473 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.606533 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.607642 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.608308 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.608317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.608784 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.609627 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.609647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.610520 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.610524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.612037 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.612050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.612816 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.612820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.613033 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.613631 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.613648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.615082 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.616830 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.616837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.616855 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.617166 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.617433 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.617437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.617439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.617596 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.617992 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.618000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.618272 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.618450 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.618809 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":140,"threads_started":1}
I20260929 14:46:35.621467 22805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:35.627081 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:35.629669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38403
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:35.633735 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.622067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52652)
0929 14:46:35.622527 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.622539 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.622607 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.623324 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.623327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.623497 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.623710 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.623718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.625398 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.625401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.626304 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.626315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.627013 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.627017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.627122 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.627800 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.627814 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.629452 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.631827 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.631830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.631832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.632007 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.632534 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.632537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.632538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.632626 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.633232 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.633235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.633323 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.633416 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.633546 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399}
I20260929 14:46:35.635449 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.622191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.98.61:33727)
0929 14:46:35.622868 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.622880 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.622905 (+ 25us) server_negotiation.cc:407] Connection header received
0929 14:46:35.622951 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.622957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.623181 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.623365 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.623846 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.625227 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.626434 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.626904 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.627000 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.629609 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.629622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.629641 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.629667 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.631724 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.632332 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.632338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.632340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.632435 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.632710 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.632713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.632714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.633066 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.633195 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.634111 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.634273 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":151,"threads_started":1}
I20260929 14:46:35.637850 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.627600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52656)
0929 14:46:35.627746 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.627763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.627857 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.628215 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.628218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.628360 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.628483 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.628490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.629743 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.629747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.631123 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.631136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.631808 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.631812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.631926 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.632348 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.632362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.633858 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.636013 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.636017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.636019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.636434 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.636785 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.636788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.636790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.636921 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.637368 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.637371 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.637469 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.637549 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.637690 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 14:46:35.638093 22805 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:35.638622 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.627677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.98.61:33727)
0929 14:46:35.627983 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.627991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.628004 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:35.628037 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.628039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.628078 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.628170 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.628735 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.629593 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.631272 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.631643 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.631758 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.634236 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.634240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.634242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.634264 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.635891 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.636586 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.636589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.636591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.636680 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.637032 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.637035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.637037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.637262 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.638279 (+ 1017us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.638351 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.638465 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":128,"threads_started":1}
I20260929 14:46:35.638909 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.638603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37432)
0929 14:46:35.638773 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.638849 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 14:46:35.639158 22756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.639392 22730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:35.640399 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.639914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37450)
0929 14:46:35.640234 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.640290 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":129,"threads_started":1}
I20260929 14:46:35.640518 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.639795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37442)
0929 14:46:35.640397 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.640445 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543}
W20260929 14:46:35.640650 22805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.641307 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.641074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37454)
0929 14:46:35.641212 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.641257 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 14:46:35.641494 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.672677 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 1
I20260929 14:46:35.673799 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.673289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37458)
0929 14:46:35.673525 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.673598 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 14:46:35.674736 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.674506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37468)
0929 14:46:35.674616 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.674670 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 14:46:35.674916 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.689936 22805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 1
I20260929 14:46:35.690925 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.690599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37472)
0929 14:46:35.690772 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.690839 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 14:46:35.691745 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.691503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37482)
0929 14:46:35.691628 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.691681 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 14:46:35.691886 22805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.750103 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 2
I20260929 14:46:35.751006 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.750686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37486)
0929 14:46:35.750857 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.750923 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 14:46:35.751734 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.751515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37502)
0929 14:46:35.751623 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.751665 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 14:46:35.751847 22730 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
W20260929 14:46:35.752506 22812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:35.752815 22812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:35.752877 22812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:35.752938 22812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:35.757236 22812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:35.757313 22812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:35.757337 22812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:35.757361 22812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:35.757390 22812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:35.760068 22805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 2
I20260929 14:46:35.760905 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.760544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37510)
0929 14:46:35.760686 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.760786 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 14:46:35.761639 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.761424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37526)
0929 14:46:35.761534 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.761580 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 14:46:35.761754 22805 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38403: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38403: connect: Connection refused (error 111)
I20260929 14:46:35.763345 22812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36199,127.0.98.61:33727,127.0.98.60:38403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:35.764449 22812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:35.765543 22812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:35.774250 22819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:35.774250 22822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:35.774319 22820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:35.774588 22812 server_base.cc:1034] running on GCE node
I20260929 14:46:35.775270 22812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:35.776532 22812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:35.777783 22812 hybrid_clock.cc:648] HybridClock initialized: now 1790693195777739 us; error 61 us; skew 500 ppm
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:35.780786 22812 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:46:35.781809 22812 webserver.cc:460] Webserver started at http://127.0.98.60:39677/ using document root <none> and password file <none>
I20260929 14:46:35.782374 22812 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:35.782444 22812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:35.782645 22812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:35.784711 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data/instance:
uuid: "d94caed1cbc54c169b31c2e48c7ba4b7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "fac283da0e1d21ae6ebbdd00d578b5a7"
server_key_iv: "73ec594fdd5c59b705a63f5197c9c443"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.785333 22812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal/instance:
uuid: "d94caed1cbc54c169b31c2e48c7ba4b7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "fac283da0e1d21ae6ebbdd00d578b5a7"
server_key_iv: "73ec594fdd5c59b705a63f5197c9c443"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.789245 22812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 14:46:35.791797 22829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:35.793040 22812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:46:35.793166 22812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "d94caed1cbc54c169b31c2e48c7ba4b7"
format_stamp: "Formatted at 2026-09-29 14:46:35 on dist-test-slave-g8hp"
server_key: "fac283da0e1d21ae6ebbdd00d578b5a7"
server_key_iv: "73ec594fdd5c59b705a63f5197c9c443"
server_key_version: "encryptionkey@0"
I20260929 14:46:35.793265 22812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:35.821354 22812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:35.824311 22812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:35.824492 22812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:35.831590 22812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38403
I20260929 14:46:35.831851 22881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38403 every 8 connection(s)
I20260929 14:46:35.832587 22812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:46:35.835573 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:35.837416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 22812
I20260929 14:46:35.837523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790692915364895-392-0/minicluster-data/master-2/wal/instance
I20260929 14:46:35.837800 392 external_mini_cluster.cc:1428] Setting key d0e8a9f024370b844491f72aff529f8d
I20260929 14:46:35.840008 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:35.850999 22730 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 3
I20260929 14:46:35.851473 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.842450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.0.98.62:36199)
0929 14:46:35.842574 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.842578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.842916 (+ 338us) server_negotiation.cc:407] Connection header received
0929 14:46:35.843038 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.843041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.843085 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.843149 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.843828 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.844296 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.845240 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.845486 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.845577 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.848109 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.848113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.848116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.848145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.849445 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.849956 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.849960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.849962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.850042 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.850314 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.850317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.850319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.850540 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.850635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.851048 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.851178 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 14:46:35.852576 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.842375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36199 (local address 127.0.0.1:59664)
0929 14:46:35.842798 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.842839 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.842953 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.843216 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.843222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.843573 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.843693 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.843699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.844407 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.844410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.845119 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.845128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.845636 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.845640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.845733 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.846172 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.846187 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.847967 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.849536 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.849546 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.849557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.849862 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.850125 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.850127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.850129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.850232 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.850640 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.850645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.850795 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.850993 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.851370 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":167,"threads_started":1}
I20260929 14:46:35.852576 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.839283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.98.60:38403)
0929 14:46:35.839702 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.839726 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.839769 (+ 43us) server_negotiation.cc:407] Connection header received
0929 14:46:35.840508 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.840533 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.841015 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.841396 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.842439 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.843667 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.844557 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.845078 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.845219 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.847665 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.847670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.847682 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.847704 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.849297 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.849738 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.849743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.849744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.849847 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.850107 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.850110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.850112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.850742 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.850891 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.851480 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.851609 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260929 14:46:35.853550 22805 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } attempt: 3
I20260929 14:46:35.855139 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:46:35.860906 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.851453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37540)
0929 14:46:35.851570 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.851586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.851659 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.852210 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.852213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.852360 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.852438 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.852443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.853322 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.853325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.854013 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.854023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.855199 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.855202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.855286 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.855790 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.855804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.857405 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.859329 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.859332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.859334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.859583 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.859905 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.859908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.859910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.859998 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.860379 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.860382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.860473 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.860561 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.860717 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 14:46:35.861357 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.851723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.60:38403)
0929 14:46:35.852020 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.852028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.852038 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:46:35.852065 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.852067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.852101 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.852176 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.852538 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.853222 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.854303 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.855061 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.855501 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.857551 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.857555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.857557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.857580 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.859198 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.859720 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.859723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.859725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.859809 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.860124 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.860126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.860127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.860289 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.860400 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.860711 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.860818 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":99,"threads_started":1}
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:35.863334 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.853969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37546)
0929 14:46:35.854069 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.854083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.854165 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.854776 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.854779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.854991 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.855170 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.855178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.856155 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.856159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.857010 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.857020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.857591 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.857595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.857716 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.858235 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.858249 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.859679 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.861741 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.861745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.861748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.861981 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.862304 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.862307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.862308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.862425 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.862825 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.862828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.862928 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.863030 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.863172 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 14:46:35.863430 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.854269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.0.98.60:38403)
0929 14:46:35.854580 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.854584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.854596 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:46:35.854626 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.854628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.854666 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.854742 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.855332 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.855953 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.857140 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.857475 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.857917 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.859817 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.859821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.859823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.859844 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.861607 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.862100 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.862103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.862105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.862175 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.862524 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.862528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.862529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.862739 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.862827 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.863099 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.863209 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260929 14:46:35.865135 22730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: Bootstrap starting.
I20260929 14:46:35.867038 22805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: Bootstrap starting.
I20260929 14:46:35.867115 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.855779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.98.62:36199)
0929 14:46:35.855966 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.855970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.855982 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:46:35.856013 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.856016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.856052 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.856129 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.856854 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.857596 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.859333 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.859639 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.859731 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.863396 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.863400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.863402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.863426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.864915 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.865396 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.865399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.865401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.865481 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.865766 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.865770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.865772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.866028 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.866235 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.866854 (+ 619us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.866969 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 14:46:35.868680 22730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:35.868707 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.855994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.61:33727)
0929 14:46:35.856096 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.856100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.856318 (+ 218us) server_negotiation.cc:407] Connection header received
0929 14:46:35.856472 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.856475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.856521 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.856568 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.857544 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.858195 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.859274 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.859631 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.859699 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.865339 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.865342 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.865345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.865369 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.866890 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.867374 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.867378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.867380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.867463 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.867763 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.867766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.867768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.868007 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.868078 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.868445 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.868564 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 14:46:35.869231 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.855627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33727 (local address 127.0.0.1:52662)
0929 14:46:35.856255 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.856268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.856377 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.856710 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.856713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.856900 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.857405 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.857414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.858331 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.858334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.859138 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.859149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.859828 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.859831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.859921 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.861168 (+ 1247us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.861181 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.865186 (+ 4005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.867020 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.867023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.867025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.867270 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.867570 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.867573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.867575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.867663 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.868150 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.868153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.868270 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.868910 (+ 640us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.869071 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565}
I20260929 14:46:35.869966 22882 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:35.870332 22805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:35.869846 22730 log.cc:824] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:35.872305 22805 log.cc:824] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:35.873383 22730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: No bootstrap required, opened a new log
I20260929 14:46:35.874161 22805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: No bootstrap required, opened a new log
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:35.876891 22730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.877233 22730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c22a3cae24712830bb246ca3ae7b3, State: Initialized, Role: FOLLOWER
I20260929 14:46:35.877631 22730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.878480 22805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.878820 22805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f497c1ecec742a1b32166666f32faa7, State: Initialized, Role: FOLLOWER
I20260929 14:46:35.878983 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.868382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.98.61:33727)
0929 14:46:35.868493 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.868497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.868652 (+ 155us) server_negotiation.cc:407] Connection header received
0929 14:46:35.868830 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.868833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.868908 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.868960 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.869803 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.870492 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.871591 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.871938 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.872007 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.875191 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.875195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.875205 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.875231 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.876979 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.877461 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.877465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.877467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.877549 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.877835 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.877838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.877840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.878052 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.878240 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.878688 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.878838 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:46:35.879204 22730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:35.879357 22805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.880707 22805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 14:46:35 dist-test-slave-g8hp krb5kdc[22649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:46:35.883021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:35.884395 22704 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "737c22a3cae24712830bb246ca3ae7b3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59674
I20260929 14:46:35.884605 22704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:46:35.884706 22704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:35.885854 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.870470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.0.98.60:38403)
0929 14:46:35.870582 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.870586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.871207 (+ 621us) server_negotiation.cc:407] Connection header received
0929 14:46:35.871334 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.871337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.871366 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.871403 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.877021 (+ 5618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.877745 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.878791 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.879076 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.879158 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.881497 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.881500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.881503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.881525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.883552 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.884016 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.884019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.884020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.884107 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.884996 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.884999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.885001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.885233 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.885302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.885586 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.885706 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 14:46:35.886000 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:35.886060 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:35.886085 22704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.886246 22886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.870400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38403 (local address 127.0.0.1:37552)
0929 14:46:35.871145 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:35.871159 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:35.871258 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:35.871507 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:35.871509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:35.871696 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:35.871782 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.871789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.877867 (+ 6078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.877870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.878669 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:35.878681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.879303 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.879306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:35.879377 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.879832 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:35.879845 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:35.881371 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:35.883704 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.883707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.883709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.883925 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.884773 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:35.884776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:35.884778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:35.884909 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:35.885356 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:35.885358 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:35.885438 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 14:46:35.885967 (+ 529us) client_negotiation.cc:240] Negotiation successful
0929 14:46:35.886113 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674}
I20260929 14:46:35.887042 22704 leader_election.cc:288] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [CANDIDATE]: Term 1 election: Requested vote from peers 3f497c1ecec742a1b32166666f32faa7 (127.0.98.61:33727), d94caed1cbc54c169b31c2e48c7ba4b7 (127.0.98.60:38403)
I20260929 14:46:35.887859 22778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737c22a3cae24712830bb246ca3ae7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f497c1ecec742a1b32166666f32faa7"
I20260929 14:46:35.888109 22778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:35.889365 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: Bootstrap starting.
I20260929 14:46:35.889451 22778 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c22a3cae24712830bb246ca3ae7b3 in term 1.
I20260929 14:46:35.889978 22679 leader_election.cc:302] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f497c1ecec742a1b32166666f32faa7, 737c22a3cae24712830bb246ca3ae7b3; no voters:
I20260929 14:46:35.890743 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.890882 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.891059 22919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:35.891140 22919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [term 1 LEADER]: Becoming Leader. State: Replica: 737c22a3cae24712830bb246ca3ae7b3, State: Running, Role: LEADER
I20260929 14:46:35.891450 22919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.892196 22882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:35.893149 22882 log.cc:824] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:35.893505 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737c22a3cae24712830bb246ca3ae7b3. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.893635 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:35.890834 22662 master_runner.cc:186] Error getting master registration for 127.0.98.60:38403: OK, instance_id { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" instance_seqno: 1790693195829744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:46:35.894134 22662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:46:35.894562 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.894760 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.894876 22737 master_runner.cc:186] Error getting master registration for 127.0.98.60:38403: OK, instance_id { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" instance_seqno: 1790693195829744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:46:35.898123 22737 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:46:35.899470 22882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: No bootstrap required, opened a new log
W20260929 14:46:35.901130 22913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:35.901212 22913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:35.901526 22922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:35.902518 22922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:35.906045 22882 raft_consensus.cc:348] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.906631 22882 raft_consensus.cc:729] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94caed1cbc54c169b31c2e48c7ba4b7, State: Initialized, Role: FOLLOWER
I20260929 14:46:35.907323 22882 consensus_queue.cc:260] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } }
I20260929 14:46:35.909543 22856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "737c22a3cae24712830bb246ca3ae7b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7"
I20260929 14:46:35.909783 22856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:35.911645 22856 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c22a3cae24712830bb246ca3ae7b3 in term 1.
I20260929 14:46:35.911970 22882 sys_catalog.cc:564] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:35.917092 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.917268 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.922135 22778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [term 1 FOLLOWER]: Refusing update from remote peer 737c22a3cae24712830bb246ca3ae7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:35.922966 22890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:35.924140 22856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [term 1 FOLLOWER]: Refusing update from remote peer 737c22a3cae24712830bb246ca3ae7b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:35.925101 22890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:35.925295 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.899010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.62:36199)
0929 14:46:35.899115 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.899119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.900936 (+ 1817us) server_negotiation.cc:407] Connection header received
0929 14:46:35.901413 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.901417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.901466 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.901515 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.908313 (+ 6798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.909046 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.911299 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.911608 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.911691 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.917060 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.917064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.917067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.917100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.918776 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.920030 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.920034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.920036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.920136 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.920478 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.920494 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.920496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.920735 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.920897 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.925009 (+ 4112us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.925154 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 14:46:35.928498 22939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:35.928603 22939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:35.929195 22885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.899386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.98.60:38403)
0929 14:46:35.899480 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.899484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.900723 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 14:46:35.901748 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.901752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.901796 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.901844 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.904578 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.905377 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.910111 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.910399 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.910478 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.920518 (+ 10040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.920522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.920525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.920552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.922068 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.926624 (+ 4556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.926628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.926631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.926745 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.927525 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.927529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.927531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.927778 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.927890 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.928919 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.929054 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 14:46:35.930521 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:35.898569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.98.61:33727)
0929 14:46:35.898688 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:35.898693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:35.901134 (+ 2441us) server_negotiation.cc:407] Connection header received
0929 14:46:35.903573 (+ 2439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:35.903577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:35.903631 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:35.903680 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:35.906486 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.907206 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.911852 (+ 4646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:35.912166 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:35.912244 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:35.922870 (+ 10626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:35.922874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:35.922877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:35.922906 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:35.924432 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.927069 (+ 2637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.927073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.927075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.927170 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:35.928202 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:35.928206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:35.928208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:35.928436 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:35.928543 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:35.930212 (+ 1669us) server_negotiation.cc:299] Negotiation successful
0929 14:46:35.930359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 14:46:35.933791 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737c22a3cae24712830bb246ca3ae7b3. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.933912 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.935604 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 737c22a3cae24712830bb246ca3ae7b3. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.935724 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.937135 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.937261 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:35.937441 22927 mvcc.cc:205] Tried to move back new op lower bound from 7334679330483478528 to 7334679330375127040. Current Snapshot: MvccSnapshot[applied={T|T < 7334679330483478528}]
I20260929 14:46:35.937728 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.937846 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.938346 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.938457 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:35.938930 22919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737c22a3cae24712830bb246ca3ae7b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c22a3cae24712830bb246ca3ae7b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36199 } } peers { permanent_uuid: "3f497c1ecec742a1b32166666f32faa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33727 } } peers { permanent_uuid: "d94caed1cbc54c169b31c2e48c7ba4b7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38403 } } }
I20260929 14:46:35.939039 22919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:35.942050 22942 mvcc.cc:205] Tried to move back new op lower bound from 7334679330483478528 to 7334679330375127040. Current Snapshot: MvccSnapshot[applied={T|T < 7334679330483478528}]
I20260929 14:46:35.944415 22941 mvcc.cc:205] Tried to move back new op lower bound from 7334679330483478528 to 7334679330375127040. Current Snapshot: MvccSnapshot[applied={T|T < 7334679330483478528}]
I20260929 14:46:35.948585 22922 catalog_manager.cc:1347] Generated new cluster ID: ef1d400d10aa4fbaaf35c95469ddd7e1
I20260929 14:46:35.948665 22922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:35.979890 22922 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:35.981334 22922 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:35.993610 22922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 737c22a3cae24712830bb246ca3ae7b3: Generated new TSK 0
I20260929 14:46:35.994686 22922 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:46:36.022318 22885 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.018867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.98.60:38403)
0929 14:46:36.019030 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.019036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.021145 (+ 2109us) server_negotiation.cc:407] Connection header received
0929 14:46:36.021378 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.021382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.021435 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.021486 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.022154 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37566: BlockingRecv error: recv got EOF from 127.0.0.1:37566 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 14:46:36.024323 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.020507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.98.61:33727)
0929 14:46:36.020625 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.020630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.023038 (+ 2408us) server_negotiation.cc:407] Connection header received
0929 14:46:36.023308 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.023312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.023365 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.023414 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.024176 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52682: BlockingRecv error: recv got EOF from 127.0.0.1:52682 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 14:46:36.025136 22734 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.019752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.98.62:36199)
0929 14:46:36.019873 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.019878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.022862 (+ 2984us) server_negotiation.cc:407] Connection header received
0929 14:46:36.023548 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.023552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.023600 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.023648 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.025020 (+ 1372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59694: BlockingRecv error: recv got EOF from 127.0.0.1:59694 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260929 14:46:36.030115 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.028424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.0.98.61:33727)
0929 14:46:36.028538 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.028542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.029090 (+ 548us) server_negotiation.cc:407] Connection header received
0929 14:46:36.029331 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.029334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.029411 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.029463 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.030005 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52698: BlockingRecv error: recv got EOF from 127.0.0.1:52698 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 14:46:36.037045 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.033836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.0.98.61:33727)
0929 14:46:36.033944 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.033949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.034530 (+ 581us) server_negotiation.cc:407] Connection header received
0929 14:46:36.034724 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.034727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.034781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.034833 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.036920 (+ 2087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52712: BlockingRecv error: recv got EOF from 127.0.0.1:52712 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 14:46:36.043256 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.041588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.0.98.61:33727)
0929 14:46:36.041701 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.041705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.042265 (+ 560us) server_negotiation.cc:407] Connection header received
0929 14:46:36.042466 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.042469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.042522 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.042574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.043123 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52720: BlockingRecv error: recv got EOF from 127.0.0.1:52720 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260929 14:46:36.051893 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.049994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.98.61:33727)
0929 14:46:36.050162 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.050167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.050802 (+ 635us) server_negotiation.cc:407] Connection header received
0929 14:46:36.051022 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.051026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.051085 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.051138 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.051758 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52726: BlockingRecv error: recv got EOF from 127.0.0.1:52726 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260929 14:46:36.064903 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.062792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.0.98.61:33727)
0929 14:46:36.062929 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.062934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.063715 (+ 781us) server_negotiation.cc:407] Connection header received
0929 14:46:36.063987 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.063991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.064049 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.064102 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.064737 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52742: BlockingRecv error: recv got EOF from 127.0.0.1:52742 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 14:46:36.085449 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.083726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.0.98.61:33727)
0929 14:46:36.083851 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.083855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.084421 (+ 566us) server_negotiation.cc:407] Connection header received
0929 14:46:36.084618 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.084621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.084674 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.084726 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.085321 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52748: BlockingRecv error: recv got EOF from 127.0.0.1:52748 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 14:46:36.121305 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.119948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.0.98.61:33727)
0929 14:46:36.120087 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.120090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.120323 (+ 233us) server_negotiation.cc:407] Connection header received
0929 14:46:36.120452 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.120456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.120546 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.120651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.121179 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52752: BlockingRecv error: recv got EOF from 127.0.0.1:52752 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 14:46:36.188231 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.187123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.0.98.61:33727)
0929 14:46:36.187301 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.187305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.187385 (+ 80us) server_negotiation.cc:407] Connection header received
0929 14:46:36.187507 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.187509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.187557 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.187627 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.188105 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52760: BlockingRecv error: recv got EOF from 127.0.0.1:52760 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 14:46:36.319339 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.318232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52764 (local address 127.0.98.61:33727)
0929 14:46:36.318376 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.318381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.318447 (+ 66us) server_negotiation.cc:407] Connection header received
0929 14:46:36.318600 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.318604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.318654 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.318737 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.319219 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52764: BlockingRecv error: recv got EOF from 127.0.0.1:52764 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 14:46:36.578379 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:36.577411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.0.98.61:33727)
0929 14:46:36.577548 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:36.577552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:36.577652 (+ 100us) server_negotiation.cc:407] Connection header received
0929 14:46:36.577742 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:36.577745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:36.577790 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:36.577860 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:36.578257 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52770: BlockingRecv error: recv got EOF from 127.0.0.1:52770 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 14:46:36.902508 22913 catalog_manager.cc:1259] Loaded cluster ID: ef1d400d10aa4fbaaf35c95469ddd7e1
I20260929 14:46:36.902707 22913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: loading cluster ID for follower catalog manager: success
I20260929 14:46:36.904387 22913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: acquiring CA information for follower catalog manager: success
I20260929 14:46:36.905647 22913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f497c1ecec742a1b32166666f32faa7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:46:36.930050 22939 catalog_manager.cc:1259] Loaded cluster ID: ef1d400d10aa4fbaaf35c95469ddd7e1
I20260929 14:46:36.930145 22939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: loading cluster ID for follower catalog manager: success
I20260929 14:46:36.931810 22939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: acquiring CA information for follower catalog manager: success
I20260929 14:46:36.933029 22939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d94caed1cbc54c169b31c2e48c7ba4b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:46:37.096767 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:37.092608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.0.98.61:33727)
0929 14:46:37.092919 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:37.092925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:37.093030 (+ 105us) server_negotiation.cc:407] Connection header received
0929 14:46:37.093166 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:37.093170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:37.093304 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:37.093403 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:37.093795 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.094324 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.095200 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.095449 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.095538 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:37.095856 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:37.096325 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:37.096456 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:37.096541 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 14:46:37.096607 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":93,"threads_started":1}
I20260929 14:46:37.104351 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:37.100229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.0.98.60:38403)
0929 14:46:37.100456 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:37.100461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:37.100653 (+ 192us) server_negotiation.cc:407] Connection header received
0929 14:46:37.100798 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:37.100801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:37.100975 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:37.101121 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:37.101326 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.101838 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.102678 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.102910 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.103017 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:37.103298 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:37.103918 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:37.104032 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:37.104152 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 14:46:37.104215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260929 14:46:37.111806 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:37.107530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.0.98.60:38403)
0929 14:46:37.107751 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:37.107756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:37.108121 (+ 365us) server_negotiation.cc:407] Connection header received
0929 14:46:37.108298 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:37.108301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:37.108347 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:37.108410 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:37.108823 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.109439 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.110801 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.111059 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.111135 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:37.111365 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:37.111446 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:37.111517 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:37.111611 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 14:46:37.111668 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 14:46:37.112385 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:37.107510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52788 (local address 127.0.98.61:33727)
0929 14:46:37.107651 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:37.107656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:37.108088 (+ 432us) server_negotiation.cc:407] Connection header received
0929 14:46:37.108279 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:37.108283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:37.108344 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:37.108404 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:37.108633 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.109421 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.111193 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.111451 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.111542 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:37.111930 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:37.112015 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:37.112086 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:37.112204 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 14:46:37.112260 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 14:46:37.113134 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:37.107530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.98.62:36199)
0929 14:46:37.107941 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:37.107946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:37.107962 (+ 16us) server_negotiation.cc:407] Connection header received
0929 14:46:37.108165 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:37.108169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:37.108341 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:37.108449 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:37.108812 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.109604 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.110469 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:37.110711 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:37.110802 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:37.112032 (+ 1230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:37.112398 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:37.112478 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:37.112843 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 14:46:37.112940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":158,"threads_started":1}
I20260929 14:46:37.116443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22662
I20260929 14:46:37.122148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22737
I20260929 14:46:37.127463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 22812
2026-09-29T14:46:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3060 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 14:46:37 dist-test-slave-g8hp krb5kdc[23031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 14:46:37 dist-test-slave-g8hp krb5kdc[23031](info): set up 2 sockets
Sep 29 14:46:37 dist-test-slave-g8hp krb5kdc[23031](info): commencing operation
krb5kdc: starting...
W20260929 14:46:38.085518 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/test-user.keytab.
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T14:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T14:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 14:46:38.210270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 14:46:38.320695 23044 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:38.321043 23044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:38.321085 23044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:38.321139 23044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 14:46:38.325430 23044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:38.325500 23044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:38.325536 23044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:38.325558 23044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:38.325588 23044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:38.331046 23044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:38.332196 23044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:38.333271 23044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:38.342145 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:38.342165 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:38.342332 23044 server_base.cc:1034] running on GCE node
W20260929 14:46:38.342165 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:38.343127 23044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:38.344254 23044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:38.345429 23044 hybrid_clock.cc:648] HybridClock initialized: now 1790693198345407 us; error 33 us; skew 500 ppm
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:38.357692 23044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 14:46:38.358747 23044 webserver.cc:460] Webserver started at http://127.0.98.62:46213/ using document root <none> and password file <none>
I20260929 14:46:38.359344 23044 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:38.359417 23044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:38.359637 23044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:38.361868 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data/instance:
uuid: "b4f01805bf80480599ee589fc089a204"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "027ae9ea41b1e00cf0c0c8386bab4ee5"
server_key_iv: "3e6769f5a20a6a53c4db9e7640bba6c1"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.362417 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4f01805bf80480599ee589fc089a204"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "027ae9ea41b1e00cf0c0c8386bab4ee5"
server_key_iv: "3e6769f5a20a6a53c4db9e7640bba6c1"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.366214 23044 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:46:38.368829 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:38.370103 23044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:46:38.370225 23044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal
uuid: "b4f01805bf80480599ee589fc089a204"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "027ae9ea41b1e00cf0c0c8386bab4ee5"
server_key_iv: "3e6769f5a20a6a53c4db9e7640bba6c1"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.370324 23044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:38.402314 23044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:38.405444 23044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:38.405611 23044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:38.412482 23044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45325
I20260929 14:46:38.412757 23111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45325 every 8 connection(s)
I20260929 14:46:38.413511 23044 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/data/info.pb
I20260929 14:46:38.416424 23112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:38.417124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 23044
I20260929 14:46:38.417251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-0/wal/instance
I20260929 14:46:38.417558 392 external_mini_cluster.cc:1428] Setting key 2850c3c06b9bca26daeae212418164cf
I20260929 14:46:38.420997 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:38.437614 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.423613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45325 (local address 127.0.0.1:41954)
0929 14:46:38.424230 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.424270 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.425220 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.425976 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.425982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.426440 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.426704 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.426718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.427854 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.427857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.428833 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.428846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.429584 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.429590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.429733 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.430321 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.430339 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.432048 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.434405 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.434414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.434423 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.434670 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.435058 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.435061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.435062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.435194 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.436022 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.436028 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.436183 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.436348 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.436840 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":182,"spinlock_wait_cycles":73728,"thread_start_us":149,"threads_started":1}
I20260929 14:46:38.437595 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.423619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.0.98.62:45325)
0929 14:46:38.424100 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.424120 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.424381 (+ 261us) server_negotiation.cc:407] Connection header received
0929 14:46:38.425572 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.425576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.425783 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.425954 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.426847 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.427732 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.428978 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.429396 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.429537 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.432224 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.432232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.432249 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.432277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.434276 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.434788 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.434795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.434797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.434940 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.435292 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.435296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.435298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.435873 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.436010 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.436443 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.436640 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":127,"threads_started":1}
I20260929 14:46:38.440209 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:38.441048 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.440706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47676)
0929 14:46:38.440855 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.440969 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 14:46:38.441259 23062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
I20260929 14:46:38.442231 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.441991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47692)
0929 14:46:38.442119 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.442172 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 14:46:38.442363 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37747: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 14:46:38.444741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37747
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 14:46:38.491631 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } attempt: 1
I20260929 14:46:38.492558 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.492187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47700)
0929 14:46:38.492377 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.492459 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 14:46:38.493433 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.493200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47710)
0929 14:46:38.493309 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.493364 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:46:38.493628 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37747: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
W20260929 14:46:38.559141 23119 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:38.559448 23119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:38.559476 23119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:38.559535 23119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 14:46:38.561832 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } attempt: 2
I20260929 14:46:38.562690 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.562351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47714)
0929 14:46:38.562522 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.562598 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 14:46:38.563476 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.563267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47718)
0929 14:46:38.563362 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.563411 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 14:46:38.563689 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37747: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37747: connect: Connection refused (error 111)
W20260929 14:46:38.564010 23119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:38.564065 23119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:38.564095 23119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:38.564122 23119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:38.564144 23119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:38.569736 23119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:38.570851 23119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:38.571902 23119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:46:38.580808 23124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:38.580911 23125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:38.580986 23119 server_base.cc:1034] running on GCE node
W20260929 14:46:38.580808 23127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:38.581617 23119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:38.582686 23119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:38.583881 23119 hybrid_clock.cc:648] HybridClock initialized: now 1790693198583864 us; error 33 us; skew 500 ppm
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:38.586913 23119 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 14:46:38.587865 23119 webserver.cc:460] Webserver started at http://127.0.98.61:43555/ using document root <none> and password file <none>
I20260929 14:46:38.588383 23119 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:38.588440 23119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:38.588625 23119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:38.590754 23119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data/instance:
uuid: "cbdfdf24897e4a5b9dc42781563b68d1"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "9777aae94409bba112a784b013f01f20"
server_key_iv: "e4c54d60b727462dd9829293907a7334"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.591285 23119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal/instance:
uuid: "cbdfdf24897e4a5b9dc42781563b68d1"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "9777aae94409bba112a784b013f01f20"
server_key_iv: "e4c54d60b727462dd9829293907a7334"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.595232 23119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:46:38.597728 23134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:38.599195 23119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:46:38.599318 23119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal
uuid: "cbdfdf24897e4a5b9dc42781563b68d1"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "9777aae94409bba112a784b013f01f20"
server_key_iv: "e4c54d60b727462dd9829293907a7334"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.599416 23119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:38.618487 23119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:38.621711 23119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:38.621901 23119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:38.629072 23119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37747
I20260929 14:46:38.629343 23186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37747 every 8 connection(s)
I20260929 14:46:38.630066 23119 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/data/info.pb
I20260929 14:46:38.631206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 23119
I20260929 14:46:38.631309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-1/wal/instance
I20260929 14:46:38.631567 392 external_mini_cluster.cc:1428] Setting key bd5d80c36e23918b388dae9a39da350a
I20260929 14:46:38.633054 23187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:38.636909 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:38.652040 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.638819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.0.98.62:45325)
0929 14:46:38.638964 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.638967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.639893 (+ 926us) server_negotiation.cc:407] Connection header received
0929 14:46:38.640902 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.640906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.640962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.641038 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.642348 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.643053 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.644490 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.644969 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.645079 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.648031 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.648034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.648036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.648064 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.649520 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.650037 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.650040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.650042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.650136 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.650620 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.650623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.650625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.650883 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.650997 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.651775 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.651896 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 14:46:38.652968 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.639205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45325 (local address 127.0.0.1:41968)
0929 14:46:38.639739 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.639786 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.640685 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.641249 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.641258 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.641707 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.642183 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.642196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.643204 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.643207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.644350 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.644360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.645147 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.645150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.645316 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.646065 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.646088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.647835 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.649641 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.649646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.649661 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.649913 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.650309 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.650311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.650313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.650480 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.651028 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.651035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.651358 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.651656 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.652028 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":120,"threads_started":1}
I20260929 14:46:38.654426 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:38.654866 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:38.660140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
/tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38543
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:38.665192 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.655309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47730)
0929 14:46:38.655470 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.655480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.655629 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.656303 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.656307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.656462 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.656556 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.656561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.657400 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.657404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.658367 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.658377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.658845 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.658848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.658950 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.659397 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.659409 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.661089 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.663563 (+ 2474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.663569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.663803 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.664159 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.664162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.664163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.664249 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.664617 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.664620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.664713 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.664855 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.665029 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 14:46:38.665180 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.655079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47724)
0929 14:46:38.655256 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.655269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.655347 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.655872 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.655874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.656063 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.656145 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.656151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.657183 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.657186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.657733 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.657741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.658328 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.658331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.658446 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.658959 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.658974 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.660515 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.663028 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.663034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.663405 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.663715 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.663719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.663809 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.664369 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.664373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.664497 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.664612 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.664762 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2}
I20260929 14:46:38.665407 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.655168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.0.98.61:37747)
0929 14:46:38.655515 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.655525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.655544 (+ 19us) server_negotiation.cc:407] Connection header received
0929 14:46:38.655588 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.655593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.655757 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.655888 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.656248 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.657088 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.657838 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.658227 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.658318 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.660690 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.660701 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.660718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.660743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.662907 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663525 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.663532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.663534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.663626 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663900 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.663903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.663905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.664267 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.664395 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.664991 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.665249 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":243,"spinlock_wait_cycles":30464,"thread_start_us":145,"threads_started":1}
I20260929 14:46:38.666304 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.655830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.0.98.61:37747)
0929 14:46:38.656113 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.656120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.656131 (+ 11us) server_negotiation.cc:407] Connection header received
0929 14:46:38.656157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.656159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.656190 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.656262 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.656656 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.657288 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.658497 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.658744 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.658827 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.661246 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.661249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.661252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.661276 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.663439 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.663900 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.663903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.663905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.664053 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.664355 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.664358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.664359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.664527 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.664607 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.664975 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.665133 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"spinlock_wait_cycles":3712,"thread_start_us":118,"threads_started":1}
I20260929 14:46:38.667474 23187 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:38.667814 23112 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:38.668299 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.667967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47140)
0929 14:46:38.668146 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.668229 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 14:46:38.668443 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.668213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47142)
0929 14:46:38.668324 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.668375 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:46:38.668519 23135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
I20260929 14:46:38.669353 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.669079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47152)
0929 14:46:38.669255 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.669301 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 14:46:38.669557 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
I20260929 14:46:38.669627 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.669378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47160)
0929 14:46:38.669516 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.669565 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 14:46:38.669739 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
I20260929 14:46:38.700770 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 1
I20260929 14:46:38.701728 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.701373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47164)
0929 14:46:38.701549 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.701628 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:46:38.702489 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.702244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47176)
0929 14:46:38.702369 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.702421 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 14:46:38.702666 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
I20260929 14:46:38.718989 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 1
I20260929 14:46:38.719902 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.719535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47186)
0929 14:46:38.719709 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.719788 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 14:46:38.720716 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.720466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47194)
0929 14:46:38.720583 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.720644 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 14:46:38.720886 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
W20260929 14:46:38.776343 23194 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 14:46:38.776748 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 14:46:38.776795 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 14:46:38.776849 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 14:46:38.777823 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 2
I20260929 14:46:38.778643 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.778326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47196)
0929 14:46:38.778475 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.778546 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 14:46:38.779306 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.779109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47204)
0929 14:46:38.779196 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.779244 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 14:46:38.779417 23112 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
W20260929 14:46:38.781584 23194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 14:46:38.781638 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 14:46:38.781656 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 14:46:38.781674 23194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 14:46:38.781702 23194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 14:46:38.787396 23194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45325,127.0.98.61:37747,127.0.98.60:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 14:00:45 UTC on ccycloud.quasar-qjvvoq.root.comops.site
build id 4680446
I20260929 14:46:38.788504 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:46:38.789040 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 2
I20260929 14:46:38.789728 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 14:46:38.789897 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.789580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47212)
0929 14:46:38.789741 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.789810 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 14:46:38.790628 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.790423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47226)
0929 14:46:38.790522 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.790568 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 14:46:38.790815 23187 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38543: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38543: connect: Connection refused (error 111)
I20260929 14:46:38.799005 23194 server_base.cc:1034] running on GCE node
W20260929 14:46:38.798964 23200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:38.798959 23201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:46:38.798964 23203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:46:38.799858 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 14:46:38.801059 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 14:46:38.802254 23194 hybrid_clock.cc:648] HybridClock initialized: now 1790693198802236 us; error 31 us; skew 500 ppm
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 14:46:38.805477 23194 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 14:46:38.806543 23194 webserver.cc:460] Webserver started at http://127.0.98.60:42877/ using document root <none> and password file <none>
I20260929 14:46:38.807183 23194 fs_manager.cc:359] Metadata directory not provided
I20260929 14:46:38.807248 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:46:38.807446 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:46:38.809566 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data/instance:
uuid: "36dd830058e241d58930a3343edf07bc"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "c0833d32b58b783bf384eff01e031f0e"
server_key_iv: "9a5ddf29456771099410ba9249e0bedd"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.810112 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal/instance:
uuid: "36dd830058e241d58930a3343edf07bc"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "c0833d32b58b783bf384eff01e031f0e"
server_key_iv: "9a5ddf29456771099410ba9249e0bedd"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.813961 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:46:38.816461 23210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:46:38.817786 23194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:46:38.817914 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal
uuid: "36dd830058e241d58930a3343edf07bc"
format_stamp: "Formatted at 2026-09-29 14:46:38 on dist-test-slave-g8hp"
server_key: "c0833d32b58b783bf384eff01e031f0e"
server_key_iv: "9a5ddf29456771099410ba9249e0bedd"
server_key_version: "encryptionkey@0"
I20260929 14:46:38.818056 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:46:38.834795 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:46:38.838320 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:46:38.838502 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:46:38.845525 23194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38543
I20260929 14:46:38.845851 23262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38543 every 8 connection(s)
I20260929 14:46:38.846504 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/data/info.pb
I20260929 14:46:38.847208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu as pid 23194
I20260929 14:46:38.847324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknwEhRJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790692915364895-392-0/minicluster-data/master-2/wal/instance
I20260929 14:46:38.847630 392 external_mini_cluster.cc:1428] Setting key eaa917189fa15211d9aec5da34293524
I20260929 14:46:38.849921 23263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:46:38.854488 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 14:46:38.863114 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.849179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.0.98.60:38543)
0929 14:46:38.849598 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.849626 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.849670 (+ 44us) server_negotiation.cc:407] Connection header received
0929 14:46:38.850557 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.850580 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.851006 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.851382 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.852137 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.853625 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.854494 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.855180 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.855336 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.858120 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.858126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.858135 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.858160 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.859920 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.860378 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.860385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.860391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.860509 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.860777 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.860780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.860782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.861319 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.861466 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.861963 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.862313 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260929 14:46:38.865664 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.856402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45325 (local address 127.0.0.1:41984)
0929 14:46:38.856818 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.856889 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.857006 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.857331 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.857336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.857778 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.857907 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.857913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.858733 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.858737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.859676 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.859687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.860134 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.860136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.860226 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.860638 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.860649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.862339 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.863933 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.863942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.863948 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.864224 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.864494 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.864497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.864499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.864590 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.865084 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.865089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.865185 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.865299 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.865462 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
I20260929 14:46:38.865918 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.856517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.0.98.62:45325)
0929 14:46:38.856675 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.856679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.856965 (+ 286us) server_negotiation.cc:407] Connection header received
0929 14:46:38.857091 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.857094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.857147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.857251 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.858039 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.858623 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.859801 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.860031 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.860132 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.862483 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.862487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.862488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.862512 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.863825 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.864322 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.864325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.864326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.864406 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.864668 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.864670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.864671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.864920 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.865006 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.865629 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.865765 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 14:46:38.866870 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 14:46:38.874915 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.865465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.0.98.62:45325)
0929 14:46:38.865934 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.865943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.865956 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:38.865990 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.865992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.866037 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.866115 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.866757 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.867426 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.868338 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.868700 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.868798 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.871434 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.871439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.871442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.871468 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.873202 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.873646 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.873649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.873650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.873734 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.873987 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.873989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.873990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.874182 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.874460 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.874558 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.874732 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":101,"threads_started":1}
I20260929 14:46:38.875851 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.867545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.98.61:37747)
0929 14:46:38.867720 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.867724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.867737 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:38.867777 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.867779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.867816 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.867880 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.868299 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.869032 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.869834 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.870088 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.870178 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.872535 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.872539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.872543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.872575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.874146 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.874601 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.874605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.874607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.874696 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.874976 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.874980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.874981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.875222 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.875301 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.875603 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.875712 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 14:46:38.876286 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.867405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37747 (local address 127.0.0.1:47742)
0929 14:46:38.867554 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.867567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.867638 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.867907 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.867910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.868069 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.868163 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.868170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.869163 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.869166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.869728 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.869736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.870188 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.870191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.870328 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.870790 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.870804 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.872115 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.874237 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.874240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.874242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.874469 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.874807 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.874810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.874812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.874888 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.875300 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.875304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.875400 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.875990 (+ 590us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.876135 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 14:46:38.876969 23263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
I20260929 14:46:38.878607 23112 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 3
I20260929 14:46:38.881954 23187 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } attempt: 3
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:46:38.886667 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.877574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.0.98.60:38543)
0929 14:46:38.877680 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.877683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.877696 (+ 13us) server_negotiation.cc:407] Connection header received
0929 14:46:38.877724 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.877725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.877756 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.877812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.878238 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.879041 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.879976 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.880240 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.880313 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.883519 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.883522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.883524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.883547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.885048 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885458 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.885461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.885463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.885546 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885811 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.885813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.885815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.886018 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.886085 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.886386 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.886499 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":2304}
I20260929 14:46:38.886672 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.876269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.98.61:37747)
0929 14:46:38.876514 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.876517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.876527 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:46:38.876557 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.876559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.876597 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.876667 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.877391 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.878256 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.879275 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.879608 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.879677 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.883169 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.883173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.883176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.883201 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.884758 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885270 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.885274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.885276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.885360 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885659 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.885662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.885664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.885872 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.886065 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.886420 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.886520 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 14:46:38.887077 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.877479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47250)
0929 14:46:38.877574 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.877588 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.877662 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.877846 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.877849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.878056 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.878143 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.878149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.879151 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.879154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.879876 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.879885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.880437 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.880440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.880517 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.880968 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.880983 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.883385 (+ 2402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.885154 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.885158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.885362 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.885634 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.885636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.885638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.885725 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.886139 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.886142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.886225 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.886803 (+ 578us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.886918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":1}
Sep 29 14:46:38 dist-test-slave-g8hp krb5kdc[23031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790693198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 14:46:38.889014 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.879015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47266)
0929 14:46:38.879194 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.879209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.879291 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.879674 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.879677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.879826 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.879910 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.879915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.881202 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.881206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.881903 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.881915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.882602 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.882606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.882694 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.883145 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.883159 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.885365 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.887485 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.887488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.887490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.887724 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.888010 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.888013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.888015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.888097 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.888538 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.888542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.888644 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.888730 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.888835 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 14:46:38.889612 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.879074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.0.98.60:38543)
0929 14:46:38.879461 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.879468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.879480 (+ 12us) server_negotiation.cc:407] Connection header received
0929 14:46:38.879510 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.879513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.879552 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.879628 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.880020 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.881098 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.882057 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.882431 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.885497 (+ 3066us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.885604 (+ 107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.885608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.885610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.885628 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.887358 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.887839 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.887841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.887843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.887925 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.888181 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.888184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.888186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.888429 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.888522 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.888805 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.888938 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"spinlock_wait_cycles":1408,"thread_start_us":123,"threads_started":1}
I20260929 14:46:38.890707 23263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: Bootstrap starting.
I20260929 14:46:38.892217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 14:46:38.893234 23112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: Bootstrap starting.
I20260929 14:46:38.893497 23263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:38.894393 23263 log.cc:824] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:38.893523 23086 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b4f01805bf80480599ee589fc089a204"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41986
I20260929 14:46:38.895594 23112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:38.896278 23112 log.cc:824] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: Log is configured to *not* fsync() on all Append() calls
I20260929 14:46:38.896620 23263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: No bootstrap required, opened a new log
I20260929 14:46:38.898425 23112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: No bootstrap required, opened a new log
I20260929 14:46:38.900780 23263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.901049 23263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36dd830058e241d58930a3343edf07bc, State: Initialized, Role: FOLLOWER
I20260929 14:46:38.900689 23112 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.901419 23112 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f01805bf80480599ee589fc089a204, State: Initialized, Role: FOLLOWER
I20260929 14:46:38.901410 23263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.901729 23112 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.901953 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.882375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38543 (local address 127.0.0.1:47272)
0929 14:46:38.882474 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 14:46:38.882486 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 14:46:38.882555 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 14:46:38.883036 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 14:46:38.883040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 14:46:38.883211 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 14:46:38.883301 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.883308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.884293 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.884297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.885163 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 14:46:38.885173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.885796 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.885799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 14:46:38.885877 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.886197 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 14:46:38.886210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 14:46:38.888030 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 14:46:38.890437 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.890441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.890443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.890710 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.896918 (+ 6208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 14:46:38.896922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 14:46:38.896924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 14:46:38.897053 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 14:46:38.901359 (+ 4306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 14:46:38.901363 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 14:46:38.901490 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 14:46:38.901609 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 14:46:38.901783 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 14:46:38.901995 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.902488 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.882539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.0.98.60:38543)
0929 14:46:38.882848 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.882854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.882864 (+ 10us) server_negotiation.cc:407] Connection header received
0929 14:46:38.882889 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.882891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.882922 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.882995 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.883426 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.884166 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.885345 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.885685 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.886039 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.888158 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.888162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.888164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.888186 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.890143 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.890883 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.890885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.890887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.896734 (+ 5847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.900954 (+ 4220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.900958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.900960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.901229 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.901345 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.902192 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.902324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260929 14:46:38.902498 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.903005 23112 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:38.903045 23263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:38.903103 23086 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:46:38.903219 23086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:38.904517 23086 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.907020 23187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: Bootstrap starting.
W20260929 14:46:38.907193 23065 tablet.cc:2369] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:46:38.908675 23086 leader_election.cc:288] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [CANDIDATE]: Term 1 election: Requested vote from peers cbdfdf24897e4a5b9dc42781563b68d1 (127.0.98.61:37747), 36dd830058e241d58930a3343edf07bc (127.0.98.60:38543)
I20260929 14:46:38.908838 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.909009 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 14:46:38.907963 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:38.909529 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:38.911283 23237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4f01805bf80480599ee589fc089a204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36dd830058e241d58930a3343edf07bc"
I20260929 14:46:38.911553 23237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:38.914757 23187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: Neither blocks nor log segments found. Creating new log.
I20260929 14:46:38.916215 23187 log.cc:824] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: Log is configured to *not* fsync() on all Append() calls
W20260929 14:46:38.922516 23297 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 14:46:38.922704 23297 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:38.920220 23194 master_runner.cc:186] Error getting master registration for 127.0.98.61:37747: OK, instance_id { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" instance_seqno: 1790693198627032 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:46:38.923532 23194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:46:38.925033 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.911066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.0.98.61:37747)
0929 14:46:38.911178 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.911182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.912102 (+ 920us) server_negotiation.cc:407] Connection header received
0929 14:46:38.912234 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.912238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.912291 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.912363 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.913299 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.914068 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.915243 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.915564 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.915644 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.918651 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.918655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.918658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.918688 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.920370 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.922905 (+ 2535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.922909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.922911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.923044 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.923921 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.923925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.923927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.924166 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.924300 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.924716 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.924840 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:46:38.916191 23044 master_runner.cc:186] Error getting master registration for 127.0.98.61:37747: OK, instance_id { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" instance_seqno: 1790693198627032 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 14:46:38.926433 23044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 14:46:38.926993 23237 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f01805bf80480599ee589fc089a204 in term 1.
I20260929 14:46:38.927498 23060 leader_election.cc:302] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36dd830058e241d58930a3343edf07bc, b4f01805bf80480599ee589fc089a204; no voters:
I20260929 14:46:38.927879 23274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:46:38.928107 23274 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [term 1 LEADER]: Becoming Leader. State: Replica: b4f01805bf80480599ee589fc089a204, State: Running, Role: LEADER
I20260929 14:46:38.928534 23274 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.930827 23274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f01805bf80480599ee589fc089a204. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.930974 23274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:38.931505 23270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.911410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47284 (local address 127.0.98.60:38543)
0929 14:46:38.911524 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.911528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.912395 (+ 867us) server_negotiation.cc:407] Connection header received
0929 14:46:38.917689 (+ 5294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.917694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.917748 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.917802 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.920897 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.921704 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.923082 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.923478 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.923549 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.927744 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.927749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.927751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.927784 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.929514 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.930032 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.930036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.930038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.930135 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.930427 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.930430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.930432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.930687 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.930808 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.931207 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.931347 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 14:46:38.935187 23187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: No bootstrap required, opened a new log
I20260929 14:46:38.935277 23309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:46:38.935784 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:38.911066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.98.62:45325)
0929 14:46:38.911178 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:38.911182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:38.925453 (+ 14271us) server_negotiation.cc:407] Connection header received
0929 14:46:38.925581 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:38.925585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:38.925656 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:38.925726 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:38.927694 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.928514 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.929691 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:38.930070 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:38.930144 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:38.931795 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 14:46:38.931800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 14:46:38.931803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 14:46:38.931831 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 14:46:38.933856 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.934345 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.934349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.934351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.934437 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 14:46:38.934733 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 14:46:38.934737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 14:46:38.934739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 14:46:38.934958 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 14:46:38.935082 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:38.935466 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 14:46:38.935605 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 14:46:38.936259 23309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:46:38.939797 23187 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.940065 23187 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdfdf24897e4a5b9dc42781563b68d1, State: Initialized, Role: FOLLOWER
I20260929 14:46:38.940446 23187 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } }
I20260929 14:46:38.943241 23161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4f01805bf80480599ee589fc089a204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cbdfdf24897e4a5b9dc42781563b68d1"
I20260929 14:46:38.943715 23187 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:46:38.943837 23160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:46:38.944232 23237 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [term 1 FOLLOWER]: Refusing update from remote peer b4f01805bf80480599ee589fc089a204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:38.944970 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:38.945449 23160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [term 1 FOLLOWER]: Refusing update from remote peer b4f01805bf80480599ee589fc089a204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:46:38.947388 23161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4f01805bf80480599ee589fc089a204 in term 1.
I20260929 14:46:38.948352 23274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:46:38.949920 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f01805bf80480599ee589fc089a204. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.950066 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.953084 23311 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f01805bf80480599ee589fc089a204. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.953245 23311 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.956408 23310 mvcc.cc:205] Tried to move back new op lower bound from 7334679342866632704 to 7334679342815150080. Current Snapshot: MvccSnapshot[applied={T|T < 7334679342866632704}]
I20260929 14:46:38.956925 23307 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.957062 23307 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:38.958446 23311 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.958559 23311 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.958900 23324 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.958998 23324 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.962541 23326 catalog_manager.cc:1259] Loaded cluster ID: df5eb0a702fe452aa351b37295bad6bc
I20260929 14:46:38.962623 23326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: loading cluster ID for follower catalog manager: success
W20260929 14:46:38.963179 23326 catalog_manager.cc:1576] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260929 14:46:38.963275 23326 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260929 14:46:38.973520 23309 catalog_manager.cc:1347] Generated new cluster ID: df5eb0a702fe452aa351b37295bad6bc
I20260929 14:46:38.973618 23309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:46:38.982127 23329 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.982296 23329 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204 [sys.catalog]: This master's current role is: LEADER
I20260929 14:46:38.983918 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4f01805bf80480599ee589fc089a204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4f01805bf80480599ee589fc089a204" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } peers { permanent_uuid: "cbdfdf24897e4a5b9dc42781563b68d1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37747 } } peers { permanent_uuid: "36dd830058e241d58930a3343edf07bc" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38543 } } }
I20260929 14:46:38.984076 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 14:46:38.984664 23322 mvcc.cc:205] Tried to move back new op lower bound from 7334679342866632704 to 7334679342815150080. Current Snapshot: MvccSnapshot[applied={T|T < 7334679342866632704}]
I20260929 14:46:39.006449 23309 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:46:39.007824 23309 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:46:39.026011 23309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f01805bf80480599ee589fc089a204: Generated new TSK 0
I20260929 14:46:39.027001 23309 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 14:46:39.058254 23268 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.055519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42008 (local address 127.0.98.62:45325)
0929 14:46:39.055666 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.055671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.057115 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 14:46:39.057315 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.057319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.057372 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.057422 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.058109 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42008: BlockingRecv error: recv got EOF from 127.0.0.1:42008 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 14:46:39.059867 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.056373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.0.98.61:37747)
0929 14:46:39.056488 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.056493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.058823 (+ 2330us) server_negotiation.cc:407] Connection header received
0929 14:46:39.059062 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.059066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.059118 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.059172 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.059742 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47770: BlockingRecv error: recv got EOF from 127.0.0.1:47770 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:46:39.061918 23270 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.060366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.0.98.60:38543)
0929 14:46:39.060486 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.060490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.060968 (+ 478us) server_negotiation.cc:407] Connection header received
0929 14:46:39.061139 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.061142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.061190 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.061239 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.061778 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47286: BlockingRecv error: recv got EOF from 127.0.0.1:47286 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 14:46:39.066677 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.064964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.0.98.61:37747)
0929 14:46:39.065078 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.065082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.065612 (+ 530us) server_negotiation.cc:407] Connection header received
0929 14:46:39.065856 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.065860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.065941 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.065996 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.066536 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47776: BlockingRecv error: recv got EOF from 127.0.0.1:47776 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260929 14:46:39.071826 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.070220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.0.98.61:37747)
0929 14:46:39.070333 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.070337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.070889 (+ 552us) server_negotiation.cc:407] Connection header received
0929 14:46:39.071084 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.071087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.071141 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.071193 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.071714 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47782: BlockingRecv error: recv got EOF from 127.0.0.1:47782 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 14:46:39.077940 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.076205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.0.98.61:37747)
0929 14:46:39.076325 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.076329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.076934 (+ 605us) server_negotiation.cc:407] Connection header received
0929 14:46:39.077152 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.077156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.077211 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.077261 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.077818 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47790: BlockingRecv error: recv got EOF from 127.0.0.1:47790 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:46:39.086218 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.084293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.0.98.61:37747)
0929 14:46:39.084462 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.084466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.085134 (+ 668us) server_negotiation.cc:407] Connection header received
0929 14:46:39.085374 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.085378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.085443 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.085496 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.086092 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47796: BlockingRecv error: recv got EOF from 127.0.0.1:47796 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260929 14:46:39.099576 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.097353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.98.61:37747)
0929 14:46:39.097572 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.097578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.098372 (+ 794us) server_negotiation.cc:407] Connection header received
0929 14:46:39.098626 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.098629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.098710 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.098788 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.099428 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 14:46:39.120406 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.118506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.0.98.61:37747)
0929 14:46:39.118651 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.118656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.119315 (+ 659us) server_negotiation.cc:407] Connection header received
0929 14:46:39.119531 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.119535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.119592 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.119643 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.120266 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47804: BlockingRecv error: recv got EOF from 127.0.0.1:47804 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 14:46:39.156080 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.154904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.98.61:37747)
0929 14:46:39.155059 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.155064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.155207 (+ 143us) server_negotiation.cc:407] Connection header received
0929 14:46:39.155339 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.155342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.155404 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.155507 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.155954 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47812: BlockingRecv error: recv got EOF from 127.0.0.1:47812 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 14:46:39.223590 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.222445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.0.98.61:37747)
0929 14:46:39.222596 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.222601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.222724 (+ 123us) server_negotiation.cc:407] Connection header received
0929 14:46:39.222890 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.222892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.222941 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.223011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.223483 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47822: BlockingRecv error: recv got EOF from 127.0.0.1:47822 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 14:46:39.355050 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.353797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47834 (local address 127.0.98.61:37747)
0929 14:46:39.353959 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.353962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.354132 (+ 170us) server_negotiation.cc:407] Connection header received
0929 14:46:39.354253 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.354256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.354310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.354384 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.354918 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47834: BlockingRecv error: recv got EOF from 127.0.0.1:47834 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 14:46:39.614410 23192 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 14:46:39.613314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.61:37747)
0929 14:46:39.613491 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:39.613495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:39.613510 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:46:39.613660 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:39.613663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:39.613712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:39.613786 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 14:46:39.614282 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47842: BlockingRecv error: recv got EOF from 127.0.0.1:47842 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 14:46:39.910913 23295 catalog_manager.cc:1259] Loaded cluster ID: df5eb0a702fe452aa351b37295bad6bc
I20260929 14:46:39.911006 23295 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: loading cluster ID for follower catalog manager: success
I20260929 14:46:39.912674 23295 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: acquiring CA information for follower catalog manager: success
I20260929 14:46:39.913908 23295 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36dd830058e241d58930a3343edf07bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:46:39.965761 23326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: acquiring CA information for follower catalog manager: success
I20260929 14:46:39.967097 23326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbdfdf24897e4a5b9dc42781563b68d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 14:46:40.133531 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:40.129099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.61:37747)
0929 14:46:40.129393 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:40.129398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:40.129596 (+ 198us) server_negotiation.cc:407] Connection header received
0929 14:46:40.129827 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:40.129830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:40.129962 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:40.130061 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:40.130379 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.130927 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.131881 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.132120 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.132233 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:40.132564 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:40.133116 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:40.133243 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:40.133302 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 14:46:40.133368 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260929 14:46:40.140513 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:40.136830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.60:38543)
0929 14:46:40.137080 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:40.137085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:40.137100 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:46:40.137255 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:40.137257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:40.137379 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:40.137491 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:40.137704 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.138245 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.139089 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.139307 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.139395 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:40.139682 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:40.140114 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:40.140224 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:40.140290 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 14:46:40.140347 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260929 14:46:40.148087 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:40.143802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.0.98.61:37747)
0929 14:46:40.144171 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:40.144175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:40.144190 (+ 15us) server_negotiation.cc:407] Connection header received
0929 14:46:40.144248 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:40.144250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:40.144295 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:40.144357 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:40.144712 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.145522 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.146754 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.147120 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.147193 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:40.147541 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:40.147640 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:40.147708 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:40.147879 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 14:46:40.147943 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260929 14:46:40.148598 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:40.143937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.98.62:45325)
0929 14:46:40.144162 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:40.144167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:40.144628 (+ 461us) server_negotiation.cc:407] Connection header received
0929 14:46:40.144755 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:40.144757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:40.144921 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:40.145184 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:40.145444 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.146180 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.147095 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.147328 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.147413 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:40.147717 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:40.148051 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:40.148153 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:40.148221 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 14:46:40.148483 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":74,"threads_started":1}
I20260929 14:46:40.149047 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 14:46:40.144068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.0.98.60:38543)
0929 14:46:40.144190 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 14:46:40.144193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 14:46:40.144896 (+ 703us) server_negotiation.cc:407] Connection header received
0929 14:46:40.145043 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 14:46:40.145047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 14:46:40.145101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 14:46:40.145167 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 14:46:40.145666 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.146235 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.147402 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 14:46:40.147708 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 14:46:40.147793 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 14:46:40.148355 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 14:46:40.148437 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 14:46:40.148493 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 14:46:40.148851 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 14:46:40.148932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 14:46:40.152179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 23044
I20260929 14:46:40.158164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 23119
I20260929 14:46:40.163370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknwEhRJ/build/debug/bin/kudu with pid 23194
2026-09-29T14:46:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3035 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6095 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284787 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST